Hi. First of all, thanks!, this is a very usefull package!. I have a problem and i think it's because the sample size in a group and the method of estimation. It is returning me "division by 0" problem. I think the part that's interesting is this:

I think it has to be something with the sample in each strata: When i drop the strata 3, that has just 1 observation, it works fine:

Thanks!

I was trying to follow the examples in the tutorials, and noticed that several of the datasets referenced weren't available. This made it tough to replicate the notebook outputs and follow along.

For example, I wasn't able to find any of the following datasets when searching through the repo:

• psu_frame.csv
• expenditure_on_milk.csv
• countycropareas.csv
• countycropareas_means.csv
• nhanes2f.csv
• nmihs_bs.csv
• nhanes2brr.csv
• nhanes2jknife.csv

Perhaps a nice way of making these available to users would be to create a datasets module like they do in the sklearn library (example):

from samplics.datasets import load_psu_frame

psu_frame = load_psu_frame()
psu_frame.head(25)

Thanks for all your hard work on this project!

Followed instructions to pip install samplics (samplics==0.3.10) and then tried some of the example code snippets from the project README page. It seems like there are newly supported features that are not yet added to the version on pip.

For example, the following throws TypeError: calculate() got an unexpected keyword argument 'precision'

import samplics
from samplics.sampling import SampleSize

sample_size = SampleSize(parameter = "proportion")
sample_size.calculate(target=0.80, precision=0.10)

And the following references a local file ("psu_frame.csv") which users do not appear to have access to in the way presented, though it appears you can import the reference dataset using from samplics.datasets import load_psu_frame.

import samplics
from samplics.sampling import SampleSelection

psu_frame = pd.read_csv("psu_frame.csv")
psu_sample_size = {"East":3, "West": 2, "North": 2, "South": 3}
pps_design = SampleSelection(
   method="pps-sys",
   stratification=True,
   with_replacement=False
   )

frame["psu_prob"] = pps_design.inclusion_probs(
   psu_frame["cluster"],
   psu_sample_size,
   psu_frame["region"],
   psu_frame["number_households_census"]
   )

As another general note, I would also add code to the examples to import all libraries needed to run each code snippet. For example, in the snippet above, I would add import pandas as pd to the top if you're going to use Panda's read_csv function.

Dear @MamadouSDiallo,

to sign off on " Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?", i recommend you fix all the minor grammatical issues. one idea is to use grammarly. just take a close look.

dear @MamadouSDiallo,

couldn't find the following:

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

check other JOSS software to see what you can add to the readme.

dear @MamadouSDiallo,

from joss. it will be cool to have some validation around correctness.

one idea = confirm some outputs w/ lumley's survey sampling package?

sample_size = SampleSize(parameter = "proportion")
sample_size.calculate(target=0.80, precision=0.10)

FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm

Hi, I have a problem when running EBLUP Area Model using my own dataset. It is always stuck on Singular Matrix error even my auxiliary variables don't have strong collinearity. Here is the error :

I have tried with some combination of auxiliary variables, but always return error. Can you explain what is the cause of this error? Thanks

Currently there isn't a way to account for response rate in the stratified sampling estimate per strata(the interface is there but not the functionality; tested it during the AAPOR meeting). I'd like to account for the individual strata in response rate and perhaps do this in a bit more flexible manner if possible. This will allow our team to use this for our sampling plan - we stratify by geographical region and the response rates which we have historical data on differ by region.

Hi, and thanks for this very useful library!

I encountered a problem when trying to perform a crosstab analysis with it, and it took me some time to understand that the problem comes from contingency tables containing zeros.

It's simpler to explain with an example, so case in point, consider the following made-up weighted dataset, with 0 individual at the intersection "Man/Other nationality":

import pandas
from samplics.categorical import Tabulation, CrossTabulation
df= pandas.DataFrame(data=
                        [["Woman", "European"]]*100 + \
                        [["Woman", "American"]]* 35 + \
                        [["Woman", "Other"]]*93 + \
                        [["Man", "European"]]*150 + \
                        [["Man", "American"]]*77,
                     columns=["Gender", "Nationality"])
df["weights"] = [1, 0.3, 8, 3, 0.7]  * 91
#Let's preview the data
print(df.head(3).append(df.tail(3)))

||Gender | Nationality | weights| |---| -------- | -----|-------- | 0 | Woman | European |1.0| 1 | Woman | European |0.3| 2 |Woman | European |8.0| ...|...|...|...| 452 | Man | American | 8.0| 453 |Man | American | 3.0| 454 |Man | American | 0.7|

If you crosstab the data, you'll notice there are 0 man of "Other" nationality :

pandas.crosstab(df["Nationality"],
                df["Gender"])

|Gender | Man |Woman| | -------- | -----|-------- | |Nationality | | |
|American | 77| 35| |European| 150| 100| |Other| 0| 93|

If I try to use samplics with it:

crosstab_samplics = CrossTabulation("count")
crosstab_samplics.tabulate(
    vars=df[["Gender", "Nationality"]],
    samp_weight=df["weights"],
    remove_nan=True,
)

it throws the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\me\env\lib\site-packages\samplics\categorical\tabulation.py", line 430, in tabulate
    - cell_est.reshape(vars_levels.shape[0], 1)
ValueError: cannot reshape array of size 5 into shape (6,1)

The problem disappear if I slightly change the dataset so the crosstab does not contain any zero anymore. I observed this problem with various datasets, so I'm almost certain the problem comes from these zero cells.

Is it supposed to throw an error like that?

I'm not certain if 1) the error simply comes from the fact that it's statistically incorrect to perform analysis with weights on crosstables containing zeros, or 2) if it's a case that the library doesn't take into account for the moment. If it's scenario 1), it might be useful to throw a more specific error message.

Anyway, here is my configuration, if it can help:

• Windows 10, Python 3.10.2 (tags/v3.10.2:a58ebcc), [MSC v.1929 64 bit (AMD64)]
• the code is executed in a virtual environment
• the following possibly relevant packages are installed:
  • numpy 1.22.3,
  • pandas 1.4.1,
  • statsmodels 0.13.2,
  • matplotlib 3.5.1,
  • scipy 1.8.0

NB: for some reason, pip install samplics --upgrade won't upgrade samplics from 0.3.12 to 0.3.13, so I had to install the newest version directly from the repo with pip install git+https://github.com/samplics-org/samplics.git . But anyway, the "cannot reshape array" message I encountered occurs in both versions, 0.3.12 and 0.3.13.

Thanks again for this library!

Excuse me, I'm getting an error in the EBLUP calculation. The error message is as follows:

RuntimeWarning: divide by zero encountered in log ll_term1 = np.log(np.linalg.det(V)).

And this is my script:

area = data_sae["id"] yhat = data_sae["cons"] X = data_sae[["ntl", 'feat_1', 'feat_2', 'feat_3']] sigma_e = data_sae["se"]

fh_model_reml = EblupAreaModel(method="ML") fh_model_reml.fit( yhat=yhat, X=X, area=area, error_std=sigma_e, intercept=True, tol=1e-8, )

This is my data collation:

id centroid_lon centroid_lat cons ntl feat_1 feat_2 feat_3 se 181 3404070003 110.399345 -7.755213 2.336299e+06 13.000000 -0.657062 0.256064 -0.718757 3.782321e+05 68 3402080003 110.367601 -7.901088 7.120323e+05 3.570000 -0.365713 0.222525 -1.224635 8.464960e+04 57 3402050003 110.322569 -7.928683 7.259114e+05 2.400000 -0.515885 0.689977 -1.196988 2.273737e-11

Add functionalities to the SampleSize class to handle sample size calculation for clusters. For example, to get the sample size for enumeration areas (EAs) or households when the final sampling unit is the person.