Hello,

I wonder if anyone has insights that when a table column is an array of struct, which is very common in real world database, for example, consider someone's profile has following schema:

  {
    "type": "array",
    "items": [
      {
        "type": "record",
        "name": "schools",
        "fields": [
          {
            "name": "school_name",
            "type": [
              "string"
            ]
          },
          {
            "name": "degree",
            "type": [
              "string"
            ]
          }
        ]
      },
    ]
  },

one example is:

    Name        |            schools                                     
    A           | [[MIT, Phd],                [MIT, master],            [MIT, bachrlo]]
    B           | [[Boston U, master], [Boston U, bachelor]]
    C           | [[CMU, bachelor]]

the most naive way I can think is break each school array as single element, like followings:

    Name        |            schools                                     
    A           |       [MIT, Phd]                           
    A           |       [MIT, master] 
    A           |       [MIT, bachrlo]

but I think this method would loss the relational information for one person, for example, if I know A study master at MIT, then it is possible that previous degree at MIT is bachelor instead of a typo (bachrlo)

So my question is:

1. how can we properly handle when the column is an array of a data structure
2. if columns have hierarchy structure, just confirm if I did correct or not: I should simply build this hierarchy structure into the PClean program (like a bayesian network ), and then just align them with hierarchy column names, am I correct here?

Thank you!

Hello, I am new to both PClean and Julia, when I tried to run the PClean experiments (julia --project=. experiments/hospital/run.jl), I got the following errors:

ERROR: LoadError: MethodError: no method matching logdensity(::AddTypos, ::String3, ::String3)
Closest candidates are:
  logdensity(::FormatName, ::Any, ::Any) at ~/lab/PClean/src/distributions/format_name.jl:33
  logdensity(::FormatName, ::Any, ::Any, ::Any, ::Any) at ~/lab/PClean/src/distributions/format_name.jl:14
  logdensity(::ExpandOnShortVersion, ::Any, ::Any, ::Any) at ~/lab/PClean/src/distributions/expand_on_short_version.jl:30
  ...
Stacktrace:
 [1] var"##663"(state#321::PClean.ProposalRowState)
   @ PClean ~/lab/PClean/src/inference/proposal_compiler.jl:101
 [2] #invokelatest#2
   @ ./essentials.jl:716 [inlined]
 [3] invokelatest
   @ ./essentials.jl:714 [inlined]
 [4] make_block_proposal!(state::PClean.ProposalRowState, block_index::Int64, config::PClean.InferenceConfig)
   @ PClean ~/lab/PClean/src/inference/block_proposal.jl:175
 [5] extend_particle!(particle::PClean.SMCParticle, config::PClean.InferenceConfig)
   @ PClean ~/lab/PClean/src/inference/row_inference.jl:70
 [6] run_smc!(trace::PClean.PCleanTrace, class::Symbol, key::Int64, config::PClean.InferenceConfig)
   @ PClean ~/lab/PClean/src/inference/row_inference.jl:146
 [7] initialize_trace(observations::Vector{ObservedDataset}, config::PClean.InferenceConfig)
   @ PClean ~/lab/PClean/src/inference/inference.jl:37
 [8] macro expansion
   @ ~/lab/PClean/experiments/hospital/run.jl:79 [inlined]
 [9] top-level scope
   @ ./timing.jl:220
in expression starting at /Users/sniu/lab/PClean/experiments/hospital/run.jl:78

my Julia version is 1.7.2, I searched online and found one similar issues (https://discourse.julialang.org/t/error-loaderror-methoderror-no-method-matching-setindex-shape-check-int64-int64-int64/18249) mentioned about new Julia need to do the broadcasting, but from error messages, I haven't figured out where to adjust it or maybe I go to the wrong direction. Any insights are appreciated. Thank you!

Sufeng

In the paper, it is implied that by default, all attributes and reference slots of a class belong to the same subproblem. However, the current implementation uses the opposite convention: that by default, each attribute or reference slot is in its own subproblem, requiring manual blocking in order to define bigger subproblems. We should consider changing this to match the paper.

Resolves #2 by changing the default subproblem-building strategy.

The result is that users can be 'more unthinking' and still get good inference, but that they may need to add subproblem hints (including to latent classes) to get acceptable performance. (I had to add hints in a couple places to recover the performance from before this change.)

Fixes to example data loading and some internal source code involving data frames due to breaking changes in the DataFrames and CSV packages (e.g. symbols versus strings being returned). Examples now running with latest versions of these packages:

(PClean) pkg> st
Project PClean v0.1.0
Status `~/dev/PClean/Project.toml`
  [6e4b80f9] BenchmarkTools v0.5.0
  [336ed68f] CSV v0.8.4
  [a93c6f00] DataFrames v0.22.5
  [31c24e10] Distributions v0.24.15
  [a98d9a8b] Interpolations v0.13.1
  [682c06a0] JSON v0.21.1
  [093fc24a] LightGraphs v1.3.5
  [1914dd2f] MacroTools v0.5.6
  [c03570c3] Memoize v0.4.4
  [91a5bcdd] Plots v1.10.6
  [f27b6e38] Polynomials v1.2.0
  [d330b81b] PyPlot v2.9.0
  [2913bbd2] StatsBase v0.33.3
  [88034a9c] StringDistances v0.10.0
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles

A good prior distribution on person names (first names, last name, etc.) -- but many other types of names including place names -- seems important for cases when it is useful to model the possibility of typos occurring in names. If we model an observed name field using a typo model, without an accurate name prior it is easy for the model to infer that a correctly spelled name is actually a version of another name but with typos introduced. I encountered this when writing a simple model of first names. Here is a minimal example:

PClean.@model CustomerModel begin

    @class FirstNames begin
        name ~ StringPrior(1, 60, all_given_names)
    end

    @class Person begin
        given_name ~ FirstNames
    end;

    @class Obs begin
        begin
            person ~ Person
            given_name ~ AddTypos(person.given_name.name)
        end
    end;

end;

query = @query CustomerModel.Obs [
    given_name person.given_name.name given_name
];

observations = [ObservedDataset(query, df)]
config = PClean.InferenceConfig(5, 2; use_mh_instead_of_pg=true)
@time begin 
    tr = initialize_trace(observations, config);
    run_inference!(tr, config)
end

Coming up with a good name prior seems like a very nontrivial task. Intuitively, if a human were performing this task, they would rely on their prior experience with names, including common spelling and translation / transliterations and knowledge of the variety closely related names with common phonetic origins, etc. A name expert would have a much more accurate name prior than a random person. Also, the statistics of names (frequency distributions, etc.) might vary widely based on the population or sub-population. One longer-term goal could be to develop an accurate name prior that represents the knowledge of a "global name expert".

Intermediate steps could be to

• Train a more accurate n-gram text model that is trained on a data set of names.

• Train or find an existing deep generative model for names.

Other steps that don't involve coming up with a name prior, but might mitigate the issue mentioned above might be:

• Come up with a more precise typo model, or an approximate typo model that somehow alleviates the issue (e.g. by upper-bounding the number of typos in a name). (This should be a separate issue).

• Use a large data set of names a directly-observed table in the model. This is equivalent to using a name prior that is a frequency-weighted distribution over these names. (A likely issue with that approach is that if a name is not observed at least once within this data set, then it might be likely to be corrected to name that is).

• Change the Pitman-Yor parameters for the underlying name table to better match statistics of real names, and more generally admit more rare names.

Also, a review of the potential consequences of a biased name prior, and approaches to reduce bias in the name priors, and/or mitigate downstream consequences of this bias, could be valuable.

The goal is to allow parameters (possibly from different classes) to be transformed before they are used as arguments to distributions. For example, linear combinations of normally-distributed parameters can still be used as the mean of a Gaussian observation. This may be useful in cases where we want to model multiple causes; for example, the rent of an apartment may be normally distributed around a linear combination of parameters representing the effects of (1) the apartment's location, (2) the apartment's size, (3) the apartment's landlord, etc.

There are (at least) two general strategies we could take for supporting this:

1. We could maintain as global state during inference a mean vector and covariance matrix for the multivariate Gaussian posterior over all MeanParameters in a program, updating it as necessary when values are observed or unobserved. Then, we could resample all MeanParameters jointly in a single blocked Gibbs update. This is probably the best approach from an inference perspective, as it fully takes into account all posterior correlations between the variables. I haven't yet worked out the math for what the update rules would be, or how expensive they'd be. A useful reference would be Dario Stein's recent paper, which describes the implementation of a language with Gaussian variables and affine transformations that supports exact conditioning, and uses the "maintain a covariance matrix and mean vector" approach: https://arxiv.org/pdf/2101.11351.pdf

2. We could perform individual Gibbs updates separately on each MeanParameter. Then when observing that N(x1+x2, sigma) == y, we think of it as an observation of x1 as y-x2 when updating x1, and of x2 as y-x1 when updating x2. This requires fewer changes to the current architecture, at the cost of possibly worse inference (more Gibbs samples are needed to converge to the same local posterior that the blocked update would have sampled from directly).

Performance of Flights model suffers without the subproblem block at

https://github.com/probcomp/PClean/commit/f51c9489dda76a6dbfd7c64fc166a5c94b13db7a#diff-2a3b7234fcda10bae8f2e3e677e2add7dc29ea841a266f8a13708c4e57ac069bR14

but it is unclear to me why this should be the case: the flight ID is always observed.

This includes

• Runtime + accuracy-over-time measurements against baseline inference algorithms (Figure 6)
• Configuration for baseline systems (HoloClean + NADEEF)
• Uncertainty-aware analysis of Rents dataset