[ { "path": "ChangeLog", "content": "Since April 2014, this changelog includes only those changes that break existing code or are especially notable. For a full list of changes, see the Git history at \nhttps://github.com/b-k/Apophenia/commits/master\n\n\n[Key, pre-March 2014: \n--Addition or improvement\n**Change that could require recoding existing code.\n!!Big.\n]\n\n October 2014\n** apop_model_stack --> apop_model_cross\n\n August 2014\n** default for apop_data_pack is .all_pages='y' (was 'n').\n** remove apop_plot_lattice, apop_plot_triangle, apop_plot_line_and_scatter, apop_plot_qq.\nFind them at https://github.com/b-k/apophenia/wiki/gnuplot_snippets .\n\n\tMarch 2014\n--Command-line tools print help should a user add a --help option.\n\n\tFebruary 2014\n!!OpenMP for threading. All calls to apop_map and friends, apop_model_draws, and others auto-thread.\n!!apop_rng_get_thread to get a thread-specific RNG, so you can thread random processes.\n\n\tJanuary 2014\n**View macro reform:\n\tApop_cols Apop_rows A contiguous set of columns or rows as an apop_data set (with names)\n\tApop_col\t Apop_row One column or row as an apop_data set\n\tApop_col_t Apop_row_t One column or row as an apop_data set, retrieved by row/col name\n\tApop_col_v Apop_row_v One column or row as a gsl_vector\n\tApop_col_tv Apop_row_tv One column or row as a gsl_vector, retrieved by row/col name\n\tApop_matrix_col Apop_matrix_row One column or row as a gsl_matrix\n**MLE methods are now strings instead of all-caps enums.\n**All blank elements of a data->text grid point to the same NUL string.\n--Add apop_model_metropolis; revise apop_update accordingly.\n--apop_draw uses metropolis to draw from any model with a log likelihood/p and where data size>1.\n**Replace all instances of output_file with output_name (GNU sed -i 's/output_file/output_name/g' *c)\n--Consolidated headers\n**apop.h no longer #includes time.h or unistd.h\n**apop_draw returns zero on success; nonzero on failure.\n**Removed the BIC-by-cells from estimation output. Added AIC_c. OLS now reports the ICs along with R^2.\n\n\tDecember 2013\n--append and replace options for apop_text_to_db\n--apop_probit bug fix\n**apop_plot_histograms now uses gnuplot's impulses, not boxes by default---handles missing zero bins better.\n**MLE path trace lists the probs/loglikelihoods in the vector of the apop_data set it produces. path is apop_data**, from apop_data*.\n**apop_data_transpose has an .inplace option, which is 'y' by default. Add .inplace='n' to existing uses.\n--siman checks constraints for the starting point.\n--Mixture models overhauled.\n--cleaned up the command-line utilities\n**removed apop_lookup.\n\n\tNovember 2013\n--rewrite apop_data_sort to allow sorting by multiple columns or text or names\n--apop_pmf now has a CDF method.\n--fixed up K-S tests.\n--removed the Swig Python wrapper from this package.\n**replaced char apop_opts.db_nan[101] with char *apop_opts.nan_string. More descriptive, easier to use.\n**apop_name_find does plain case-insensitive search; no regexes.\n\n\tOctober 2013\n**all built-in models (apop_ols, apop_dirichlet, ...) are now apop_model* (ptr-to-struct), from apop_model (plain struct).\n**apop_estimate and apop_copy take in an apop_model* instead of plain apop_model.\n**printing no longer part of the apop_model struct; uses a vtable.\n\n\tSeptember 2013\n**change vbase, m1base, m2base ==> vsize, msize1, msize2\n**Estimate returns void (was apop_model*)\n--vtable mechanism improvements\n**Remove score, predict, and parameter_model from the apop_model object; use the vtable mechanism.\n**Upgrade model p, ll, cdf, constraint to return long double (was double)\n**consolidate vector_var and vector_weighted_var. same with cov, mean, weighted_skew, and weighted_pop. Users just have to replace apop_vector_weighted_var with apop_vector_var.\n**removed deprecated.h entirely.\n--apop_data_add_named_elmts puts new data in the vector, not the matrix, because it is intended for a list of scalars (==a vector). If you use apop_data_get(infodata, .rowname=\"statistic name\") then you'll be able to retrieve the element either way.\n**removed apop_line_to_data and apop_line_to_matrix. Use apop_data_fill and apop_data_falloc.\n\n\tAugust 2013\n--apop_map(_sum) properly threads data-row mappings. .inplace='v' to return NULL.\n**Remove apop_settings_alloc, apop_settings_group_alloc\n**Change Apop_row to return an apop_data set, not a vector (for which use Apop_matrix_row).\n**Apop_settings_set sets model->error='s' on error, instead of returning.\n**Add a .want_path='y' setting to your apop_mle_settings group, and I'll put a list of the points tried by the optimizer in an apop_data set named path (in the settings group); see documentation for details. Remove the former trace_path mechanism.\n**removed apop_(vector|matrix)_increment. Use, e.g., *gsl_vector_ptr(v, 7) += 3; or (*gsl_vector_ptr(v, 7))++.\n**Some mu and sigmas => μ and σ\n**Removed apop_settings_alloc, apop_settings_group_alloc\n**Change Apop_row to return an apop_data set, not a vector (for which use Apop_matrix_row).\n\n\tJune 2013\n--replaced makefile in base directory with ./configure.\n--version number now equals build date.\n**name->title is a ptr; name->column => name->col\n**removed apop_strip_dots; it's up to the user to give reasonable names for the db.\n\n\tMay 2013\n--jacobian transformations\n--Apop_model_copy_set to copy a model and add a settings group at once\n--mixture models\n--data-data composition\n--added apop_model_draws\n!!vtables, allowing for more functions with special cases for certain model(s) outside of the model object itself.\n\n\tApril 2013\n--plugged some memory leaks\n--default tolerance for MLE is much finer (1e-5).\n--Added apop_text_fill\n**Finally removed support for gsl_histograms, including the apop_histogram model. This cut has been promised for about four years now. Use the apop_pmf instead.\n**apop_data_to_bins no longer modifies the input data in place. It now makes a copy and modifies the copy.\n**Removed apop_crosstab_to_pmf. There's a version at https://github.com/b-k/Apophenia/wiki/Crosstab-to-PMF for matrices.\n**Removed apop_vector_to_array. If your array has stride 1, use your_array->data; else write a for loop to copy out the data.\n**Removed apop_array_to_matrix.\n\n\tMarch 2013\n**apop_text_paste now prints the pasted string at verbosity level 3 (formerly 2).\n\n\tFebruary 2013\n--Starting_point in Bayesian updating no longer does anything, but it was never significant to begin with. Added some verbosity options to apop_update.\n\n\tJanuary 2013\n--Logit regression much smarter about picking a starting point.\n--Defaults for simulated annealing try 1600x fewer points. Prior settings were overkill.\n--configure.ac checks for native asprintf and uses it if it is present\n**Removed apop_db_merge and apop_db_merge_table. Get them from http://modelingwithdata.org/arch/00000141.html .\n**Removed apop_matrix_fill; use apop_data_fill\n**Removed apop_array_to_data.\n**Removed apop_matrix_correlation; use apop_data_correlation\n**Deprecated apop_rank_compress; use apop_data_to_dummies(..., .keep_first='y').\n--apop_model_stack\n\n\tDecember 2012\n--Added an apop_pmf_settings group, eliminating a couple of hacks (e.g., see October 2012).\n--faster read-in of text files\n--Exponential model uses data in both the vector and matrix parts of the apop_data input\n--transformation to generate mixtures (i.e., linear combinations) of models.\n**apop_data_transpose now transposes the text element as well as the matrix (by default).\n\tUse apop_data_transpose(your_matrix, .transpose_text='n') to replicate the previous behavior.\n--removed Autoconf pkg-config macros, because Autoconf no longer needs the help.\n--writing apop_data to DB uses prepared queries where possible => much faster.\n**It is now up to you to put apop_query(\"begin\"); / apop_query(\"commit\"); wrappers around\n\twriting of tables to the database.\n\n\tNovember 2012\n--apop_vector_unique_elements, apop_data_to_dummies, and apop_data_to_factors handle NaNs\n\tbetter; put them at the end of the sort order.\n!!Finally added an .error element to apop_data and apop_model structs, thus simplifying\n\terror-checking.\n--Apop_stopif macro, rendering the Apop_assert family largely obsolete (so if you're using\n\tthem in your own work, consider them deprecated...).\n--Where the assert macros used to abort() on errors, they now send signal(SIGTRAP), making\n\tdebugging a little easier. Most host systems force an abort on SIGTRAP anyway.\n\n\tOctober 2012\n**Removed apop_strcmp. If you still need it, this macro is basically equivalent:\n #define apop_strcmp(a, b) (((a)&&(b) && !strcmp((a), (b))) || (!(a) && !(b)))\n--clean up of parameter-fixing model transformation.\n--split off multiple imputation variance code.\n**Removed apop_vector_grid_distance. Use apop_vector_distance(v1, v2, .metric='M');\n--If apop_pmf.dsize==0, apop_pmf.draw returns a row number, not the data in that row. This will change shortly.\n--Logit draw function, akin to apop_ols.draw. Both will change shortly.\n--apop_data_to_factors now auto-allocates a matrix if need be (because it always auto-allocated a vector). \n\n\tSeptember 2012\n--\\0 in text files counts as white space\n--fixed counting bug for text files with , sequences.\n\n\tJuly 2012\n--some MLE cleanup\n--fixes to apop_rake\n--apop_logit.score fixed\n\n\tJune 2012\n--The sample kurtosis calculation is still more precise.\n\n\tMay 2012\n--Autotools improvements. Use the standard 'make check' instead of the ad hoc 'make test'.\n!!Set apop_opts.stop_on_warning='n' to never abort() on any type of error. E.g., GUIs that\n\tshould never halt will use this. Default is still to halt on errors, because that's\n\tmost useful for interactively developing numeric analyses.\n--Use apop_opts.log_file=fopen(\"yourlog\", \"w\") to divert the warnings/errors from stderr into yourlog.\n--Some formerly void functions now return an int, to return an error code. E.g.,\n\tapop_opts.stop_on_warning='n';\n\tif (apop_data_set(data, row, col)) printf(\"Error! Nothing was set.\\n\");\n--Fixed a memory leak in simulated annealing.\n--Apop_data_row and apop_data_set_row handle row names\n\n\tMarch 2012\n--bug fix in apop_text_to_data when input file has no names.\n--probit dlog likelihood isn't implemented for N>2; now acknowledging this.\n--added apop_data_get_factor_names\n--apop_(vector|matrix)_(map|apply) now accepts NULL input.\n\n\tJanuary 2012\n**Removed apop_matrix_var_m, which nobody was using.\n--bug fix in apop_vector_distance for Ln norms where n is odd.\n--reading data from text files rewritten; much more robust. \n **A space is no long a default delimiter. Use apop_opts.input_delimiters=\"| ,\\t\" to restore old default.\n **apop_opts.db_nan is no longer a regex; I just do a case-insensitive comparison.\n\n\tDecember 2011\n--apop_model.textsize is now a size_t instead of an int.\n--apop_update accepts likelihoods with no pre-set parameters \n\n\tNovember 2011\n--moved to Github; some changes to structure and documentation to accommodate. \n**apop_ols.predict didn't do the OLS shuffle if the input has no vector; this was anomalous.\n\n\tOctober 2011\n--apop_text_paste added.\n**apop_multinomial and apop_binomial overhauled. No longer accepting Bernoulli draws as input.\n--standardization: make docs => make doc\n\n\tSeptember 2011\n--`query turned up a blank table' warning turns up when apop_opts.verbose >=2. (used to be >=1)\n--apop_t_test and apop_paired_t_test are quieter---no intermediate results until apop_opts.verbose >=2.\n**apop_opts.db_name_column now has a blank default (instead of the SQL-specific and potentially surprise-inducing \"rowname\").\n\n\tAugust 2011\n--Bootstrap/Jackknife are better with text\n**apop_data_memcpy used to reallocate memory for the text and names elements; use apop_data_copy if you want allocation done for you.\n\n\tJuly 2011\n--Apop_data_rm_rows now accepts a test function as well as a fixed list of rows to drop.\n\n\tJune 2011\n--apop_crosstab_to_db handles missing labels and NaNs better.\n**apop_matrix_to_db removed (as promised a few years ago). Use apop_matrix_print(yourdata, \"tabname\", .output_type='d').\n**F-test defaults now match ANOVA tradition.\n--documentation script doesn't use GNU extensions to awk; should now be POSIX-standard.\n\n\tMay 2011\n**apop_map returns a data set with an allocated/filled vector when not called with .inplace='y'. Before, it had been making a full copy, which is idiosyncratic.\n--apop_rake accepts a weights column.\n--apop_anova uses variadic arguments for a marginally nicer interface (and better argument checking).\n**apop_data_to_dummies tries to give nicer labels. You may have to recode things if you relied on the old labels.\n\n\tMarch 2011\n--Header files have been merged. A few long files is as easy to grep as a multitude of\n\tnearly one-line files. If you #include instead of the individual headers,\n\tthen this shouldn't affect you. Due to redundancy, compilation with gcc takes 3% longer.\n\n\t0.99 February 2011\n!!The apop_PMF model has more support:\n\t--New supporting functions: apop_data_pmf_compress, apop_model_to_pmf\n\t--functions that took apop_histogram models now take apop_pmfs as well:\n\t\tapop_test_chi_squared_goodness_of_fit, apop_test_kolmogorov\n\t--Consider the apop_histogram to be deprecated. Only two associated functions were removed; see below.\n\t**apop_histogram_plot is removed. Replace with:\n\t fprintf(apop_opts.output_pipe, \"plot '-' using 1:3 with boxes\\n\");\n\t\tapop_model_print(hist);\n\t\tfprintf(apop_opts.output_pipe, \"e\\n\");\n\t**apop_histogram_print was a bad idea to begin with, because it basically replicates\n\t gsl_histogram_fprintf. Use apop_model_print(your_histogram), which calls gsl_histogram_fprintf, \n\t\tor call that function directly. The only difference: the GSL function prints \n\t\t[start of bin] [end of bin] [value]\n\t\tand apop_histogram print showed\n\t\t[start of bin] [value]\n\n\tDecember 2010\n**apop_maximum_likelihood no longer calls apop_prep. If you want that, use apop_estimate.\n--apop_text_to_db lets users specify types and keys.\n--deleted some obsolete/deprecated items: apop_error, apop_multinomial_settings\n--apop_data_split retains text when splitting by rows; still ignores it when splitting by columns.\n\n\tNovember 2010\n--apop_listwise_delete uses apop_opts.db_nan to check for missing data in the text part of the input data.\n--apop_data_split handles names\n\n September 2010\n**Multinomial distribution sets N to be the length of the row (a single observation)\n rather than the size of the full data set. Added apop_multinomial.parameter_model method\n for testing purposes.\n\n August 2010\n** What was apop_assert => apop_assert_c; what was apop_assert_s => apop_assert. Their\n arguments are slightly different, and the thing that was asserted no longer prints along\n with the message you chose.\n--verbosity defaults to 1. Queries print at verbosity >=2.\n--apop_data_to_db writes the weights.\n--Iterative proportional fitting, aka raking. \n\n**apop_text_add now frees the contents of cell in the text grid that you are about to\n overwrite, thus preventing memory leaks without effort from the user. If your existing\n code has other pointers to the string in that text cell, you'll have to replace the now\n string-freeing apop_text_add with asprintf(&(your_dataset->text[row][col]), \"your string\").\n\n July 2010\n** apop_regex now gets all matches when you pull substrings. Each row of the text grid\n is a match, and if you have multiple substrings, each match's substrings will be\n along the columns. May require recoding because the substrings used to be along\n the rows; just switch the indices.\n\n June 2010\n** Removed the apop_rank settings group, and thus all the code related to it. It was just\n the wrong place to do this. Added apop_data_rank_expand to convert rank data to\n what the various models typically expect. This is another step for some users and\n could be a problem if the counts get into the billions, but it still makes more sense\n than rewriting every model twice.\n\n May 2010\n**apop_data_prune_columns_base now takes in a list of strings terminated by a NULL, not by\na zero-length string.\n--apop_data_get_row lets you pull a view from a data set. [this was briefly the apop_data_row]\n ==>apop_data_set_rows, apop_data_rm_rows\n ==> apop_data_listwise_delete is fifty lines shorter.\n--apop_parts_wanted_settings: fixes some infinite loops (est needs parameter models ->\n p.m. bootstraps for variances -> bootstrap runs estimate -> repeat) and allows\n just-the-parameters estimation when you want it.\n--cleaned up build system, including an added RPM spec file\n\n April 2010\n--apop_t_distribution now has three parameters: mean, std dev, df. That is, it is based on un-normalized data.\n**apop_random_int and apop_random_double removed for not being particularly useful.\n\n March 2010\n**The apop_predict special case for when all data is non-missing was a bit too special,\n and has been eliminated---you now have to specify the first column as NaN yourself.\n E.g., Apop_col(data, -1, to_nan); gsl_vector_set_all(to_nan, GSL_NAN);\n This will make things more predictable, and save you if(!has_nans)... else... kind of statements.\n**Removed the prepared element of the apop_model.\n**apop_model_prep ==> apop_prep for consistency with other apop_model dispatch functions. apop_model_prep left for now as an alias in deprecated.h\n!!apop_parameter_model: a method for getting the distribution of a parameter.\n **Moved OLS-family test stats (pval, qval, whatever) to a page of your_estimate->info. It won't be there for long either.\n--settings macros let you use lowercase, thus entirely ignoring that they're macros.\n **apop_settings_rm_group function, which you were probably not using, changed to apop_settings_remove_group; apop_settings_get_group function => apop_settings_get_grp. Having a macro and a function that differed by a question of case was a bad idea to begin with.\n\n February 2010\n!!Overhauling the output from estimations; pardon our dust. \n--Added CDF method to the apop_model, including apop_cdf dispatch method and default via random draws.\n**Defininitvely removed the residual, covariance, and llikelihood elements from the\n apop_model struct. The first two will be pages appended to the data and parameters,\n respectively, and the last will be in the Info page appended to the parameters.\n**Renamed apop_ls_settings (least square) to apop_lm_settings (linear model) \"s/apop_ls/apop_lm/g\" should work.\n**Sundry lists of scalars, like the R^2 table and the estimation routine's info table put the data in column zero, not column -1. In the next bullet point you'll see how this simplifies retrieval.\n**Added an info element to the apop_model--> more shuffling of auxiliary info. \n --Find results like the log likelihood or AIC via, e.g., apop_data_get(your_model->info, .rowname=\"log likelihood\");\n **Find the predicted/residuals via apop_data_get_page(your_model->info, \"Predicted\");\n This means that the input data set is read-only again. \n --Find the parameter covariances via apop_data_get_page(your_model->parameters, \"Covariance\");\n **apop_estimate_coefficient_of_determination takes in the model again. Just replace est->parameters in your argument with est. apop_ols calls this fn automatically now [apop_data_get(your_model->info, .rowname=\"R sq\")], so you probably aren't even calling it anymore.\n**apop_data_add_named_elmt now writes to the zeroth element of the matrix, not the vector.\nSo instead of apop_data_get(data, .rowname=\"R squared\", -1), just go with apop_data_get(data, .rowname=\"R squared\"). This affects many of the elements of the info-type matrices.\n--apop_data_pack, apop_data_unpack, apop_ml_impute, apop_map offer an option to use all pages. \n\n 0.23 January 2010\n**expected_value element of the model renamed predict; made coherent across models.\n!!apop_data set now has a ->more pointer to an additional apop_data set, e.g., for data + covariances or predictions + confidence intervals.\n--apop_ml_imputation renamed to apop_ml_impute. \"#define apop_ml_imputation apop_ml_impute\" retains noun-form name, but consider it deprecated.\n!!apop_estimate now copies, preps, then estimates. Estimate method of apop_model struct can thus assume the copy/prep step has been done;\nprobably should not do these itself. As a side-effect, apop_maximum_likelihood's second argument is now a apop_model* (used to be apop_model).\n--apop_regex and apop_strcmp, for easier searching through your info pages.\n--minor rewording of COPYING2.\n**Because the Predicted table is now part of the parameter set, not the model,\n apop_estimate_coefficient_of_determination now takes in the parameter set, not the model. Just replace est in your argument with est->parameters.\n**apop_multinomial_probit folded into apop_probit, where it should've been all along.\n Regex for the fix: \"s/apop_multinomial_probit/apop_probit/g\"\n\n December 2009\n--apop_strcmp\n--apop_loess model: 3,500 lines of code from the netlib archive, lovingly restored.\n\n November 2009\n--apop_rm_columns bug fixed by Birger Baksaas.\n--apop_text_to_db attaches numeric affinity to sqlite3 columns, making numeric comparisons easier.\n--apop_histogram_model_reset's first argument is now \"base\" instead of \"template\", as a concession to C++ users.\n\n\tSeptember 2009\n--Many minor changes, mostly regarding adding optional arguments.\n--Dirichlet model\n--Output functions now take a consistent set of specs regarding to where they will write. You no longer have to use the global apop_opts settings if you don't want to.\n\n\tAugust 2009\n--apop_map and apop_map_sum. Reworks the apop_(map|apply) system to be more flexible but a little more complex.\n-apop_(data|matrix|vector)_fill is now more robust---no more int vs float issues.\n**Removed apop_count_cols\n--Default univariate RNG, if you don't have one: Adaptive rejection markov chain sampling\n**.use_covar and other such settings now take 'y' or 'n', not 0 or 1.\n--new macro Apop_settings_set = Apop_settings_add, but makes more human sense\n--numeric covariance, formerly maligned, now works.\n\n\tJuly 2009\n--multivariate gamma, log-gamma.\n--t, F, chi^2, Wishart distributions, for description [and Bayesian\nupdating]\n--apop_matrix_to_positive_semidefinite and apop_matrix_is_positive_semidefinite \n--bug fixes\n--Apop_model_add_group replaces Apop_settings_add_group, and is much more easy to work with.\n\n\tJune 2009\n--More variadicized functions\n\t--notably, apop_estimate is much more useful\n--apop_opts.version.\n--apop_(vector|matrix|data)_stack have an inplace option, making stacking inside a for loop easier.\n--apop_test convenience function\n--more autoconf macros ==> some compilation hacks now done right\n\n\tMay 2009\n--mysql functions slightly cleaned up\n--apop_opts.db_user and apop_opts.db_pass for mysql.\n!!Functions that take lots of basically optional inputs, like apop_text_to_db, now use some designated initializer magic to let the user rearrange or omit inputs.\n**apop_dot also now allows designated initializers, which breaks\n(only) calls of the form apop_dot(a_vector, a_matrix, 't'). Replace with\napop_dot(a_vector, a_matrix, 0, 't') or apop_dot(a_vector, a_matrix,\n.form2='t')\n--With optional inputs, some functions now handle RNGs for the lazy user ==>added apop_opts.rng_seed\n--apop_vector_distance is much more versatile\n**Removed apop_matrix_summarize. Too much like apop_data_summarize. Just\nreplace every instance of apop_matrix_summarize(m) with apop_data_summarize(apop_matrix_to_data(m)).\n\n\n April 2009\n--sample moments are now mega-accurate---possibly the most unbiased estimators in code today.\n!!Python interface via swig\n\n\tMarch 2009\n--apop_matrix_realloc, apop_vector_realloc\n--sqlite queries no longer rely on a temp table ==> faster\n--fixed bugs in apop_table_exists making queries fail in Cygwin.\n\n\tJan/Feb 2009\n--Added more tests; some cleanup in test.c\n--Binomial distribution looks in both the data set's vector and matrix\n\n\tDecember 2008\n--When writing x=infinity to a db table, I now write 'inf' to the db, instead of breaking. SQLite has no standard here.\n\n\tOctober 2008\n--bug fixes to new apop_data_show\n--bug fixes to apop_bernoulli.p\n--apop_update tweaks\n\n\tSeptember 2008\n--Documentation overhaul\n--apop_data_show is much more screen-friendly. Keep using\napop_data_print for more machine-readable and less fixed-width output.\n**apop_plot_histogram now takes in a vector, bin count, and name of output. This is what it did in the first half of the year.\n The current version of apop_plot_histogram, which acts on a histogram model, is renamed to apop_histogram_plot.\n\n\tAugust 2008\n--Constraints in ML work better.\n**Overhaul of some discrete choice models\n\t--Added tests for the probit and logit.\n\t--fixed a bug revealed by the tests\n\t**the first choice has a fixed value of zero.\n\t**You'll probably need to call Apop_category_settings_add before estimating \n\t\tyour model, unless the outcome choice variable is the 0th column of the matrix.\n\t--more to come, e.g., multinomial probit will be merged with ordered probit.\n-Adding a settings group of a given type when that group already exists used to induce an error; now the old type is replaced with a clean default.\n--bug fix for apop_test_fisher_exact on non-square matrices\n--apop_settings_add and company do more work in functions and less in macros.\n--removed the settings_ct element of the apop_model; using a sentinel at the end of the array instead.\n--Slightly improved reading of text files.\n--Bootstrap/jackknife act on models with parameters in both matrix and vector form.\n\n\n July 2008\n--Guts of apop_plot_histogram now use the apop_histogram model. \n** Also, it no longer normalizes the histogram to integrate to one by\n default. You need to explicitly request this via\n apop_histogram_normalize\n--apop_plot finally deleted.\n--apop_histogram_plot deleted; use apop_plot_histogram.\n--Added apop_vector_skew_sample, apop_vector_kurtosis_sample.\n\n\n\tMay 2008\n--apop_settings_rm_group added.\n--mysql interface has the beginnings of support for multiple\n\tsemicolon-separated queries in one call.\n--apop_histogram_refill_with_model ==> apop_histogram_model_reset;\n\tapop_histogram_refill_with_vector ==> apop_histogram_vector_reset.\n\n\tApril 2008\n--apop_dot handles names.\n--apop_t_test now behaves correctly when one vector is of length 1.\n--some improvements/fixes when dealing with mySQL.\n--apop_sv_decomp renamed to apop_matrix_pca. Minor changes so that it correctly works as such.\n--apop_text_to_(db|data) handles column names like it used to, which\n works better. Also a few other fixes for odd situations.\n\n\tMarch 2008\n--Various improvements in reading in from text.\n\t-- a, \"b, c\", d will now correctly read in as three elements: a; then \"b, c\"; then d\n\t-- a,,b,c reads as a, NAN, b, c.\n\n\tFebruary 2008\n--Some of the header references didn't work for a fresh install.\n--bug fixes, esp. with apop_test_kolmogorov\n--added convenience fn apop_data_transpose\n\n\tJanuary 2008\n--Apop_assert, which streamlines the use of apop_error (thus shrinking the code base by 2%).\n--apop_OLS now has a log likelihood (also shuffled some of the code around)\n--bug fix in apop_binomial.p.\n**More name reform: apop_correlation_matrix --> apop_matrix_correlation; apop_data_correlation_matrix --> apop_data_correlation; apop_covariance_matrix --> apop_matrix_covariance; apop_data_covariance_matrix --> apop_data_covariance.\n--apop_count_(rows|cols)_in_text are now static functions and removed from the documentation.\n--Removed apop_random_beta, which had been set as deprecated earlier. Use apop_beta_from_mean_var.\n**Removed apop_vector_isnan---just use apop_vector_map_sum(your_vector, isnan)\n**Removed apop_vector_finite---just use apop_vector_bounded(your_vector, INFINITY)\n--For your convenience, added Apop_settings_alloc() macro.\n\t**apop_histogram_params ==> apop_histogram_settings\n\t**apop_kernel_density_params ==> apop_kernel_density_settings\n\t[The settings/params distinction is in some ways arbitrary anyway.]\n--bug fix in apop_mle.c: wasn't copying output parameters to the estimated model in some cases.\n--As part of name reform, all function names are being switched to\nlower-case throughout, so apop_ANOVA ==> apop_anova. Am keeping the old\nforms via macros. Notice also the non-yelling macro capitalizations above, such as Apop_assert.\n!!**Revised the settings for the apop_model. model_settings and\nmethod_settings are out, replaced by a much more organized single list\nof settings.\n--added the apop_lookup command-line program.\n**Renamed the apop_multinomial_logit model the apop_logit, because the\nbinary logit is a special case that requires no special handling.\n**Reversed the signs on the probit coefficients, to better conform to\nthe norm.\n\n\n\tDecember 2007\n--added apop_vector_moving_average\n--apop_model now has prep and print methods.\n**apop_p, apop_log_likelihood, apop_score now take a pointer to an apop_model, not the model itself.\n--apop_multinomial_probit model\n--When a data set has matrix and vector, apop_dot accepts a 'v' to use the vector.\n--apop_plot_lattice produces a more attractive (and standard-form) plot.\n--apop_data_print works much better now.\n**apop_model no longer requires your data input to be const. It probably\nwill be const, but it's not the interface's place to dictate that.\n**apop_data_unpack no longer allocates a new data set, but writes to an input data set assumed to be of the right size.\n--added apop_ANOVA to produce one- or two-way ANOVA tables from the database.\n**apop_test_ANOVA renamed to apop_test_ANOVA_independence to create a little more cognitive distance.\n--apop_data_text_to_factors\n--APOP_COL_T and APOP_ROW_T macros, to pull a column or row by name\n--apop_beta_from_mean_var produces a beta-distributed model with the right (alpha, beta) parameters. \n**Thus, apop_random_beta is now marked as deprecated.\n**apop_x_prime_sigma_x removed, on grounds of being silly. If you want it back, see model/apop_multivariate_normal.c, where it is now a static function.\n**apop_qq_plot --> apop_plot_qq\n--Multinomial logit model (and the probit now has names).\n**Revised the bin-syncing methods. apop_vectors_to_histogram and apop_model_to_histogram are now out; apop_histogram_refill_with_vector and apop_histogram_refill_with_model are in. \n**Also removed apop_model_test_goodness_of_fit as redundant. Just produce a histogram, use the above refill functions, and send your two histograms to apop_histograms_test_goodness_of_fit. If you do this often, you can write a convenience function to do that as quickly as I could.\n**apop_vector_replace and apop_matrix_replace are redundant---just use apop_(vector|matrix)_apply.\n--The covariance matrix is now produced via the derivative of the score function at the parameter. I follow Efron and Hinkley in using the estimated information matrix---the value of the information matrix at the estimated value of the score---not the expected information matrix that is the integral over all possible data.\n\n\tNovember 2007\n--added apop_model_copy_set_string to get a copy of a model whose\nmodel_settings is just a string.\n**Thanks to this, folded all of the _rank versions of models into their\nbase models. Set model_settings to \"r\" to use the rank version.\n--The default MLE method is now the Nelder-Mead Simplex algorithm,\ninstead of the Fletcher-Reeves conjugate gradient. This is more\nconservative.\n--apop_(vector|matrix|matrix_all)_map_sum to get the sum of a function\napplied to a vector. E.g., find count of NaNs with apop_vector_map_sum(v, isnan);\n--apop_logit bug fix.\n\n\tOctober 2007\n\n--apop_estimate now defaults to using MLEs, meaning you don't have to explicitly specify an estimate method for MLE models. \n--apop_crosstab_to_db reads both the matrix and text elements of the input apop_data set. \n--apop_system convenience function, to make C feel more like a scripting language. \n--Added some SQLite functions for mySQL compatibility: var_samp, var_pop, stddev_samp, stddev_pop, std. \n--Probit patched to not NaN for very unlikely parameter/data combinations. \n**apop_estimate_restart takes two models, rather than one model and some haphazard settings. \n--apop_plot_query no longer forces you to use the -d and -q switches to specify the database and query. \n**The two places that use regular expressions: apop_opts.db_nan and the search for a name via apop_data_get/set... use case-insensitive EREs. Before I'd been using BREs, which nobody likes. \n-- stops the MLE searches, prints output, and continues the program. Especially useful for simulated annealing. [GDB tip: use the command: signal SIGINT ] \n--apop_mle_fix_params debugging: gradients work now. \n--apop_test_ANOVA added, to test the null hypothesis that all cells of a crosstab are equally likely. \n**apop_multivariate_normal_prob removed. Use the apop_multivariate_normal model and its .log_likelihood, .p, .draw, et cetera. \n**sed -i -e \"s/apop_OLS_params/apop_ls_settings/g\" -e \"s/apop_mle_params/apop_mle_settings/g\" *.c *.h\n\n\tSeptember 2007\n--The optimization methods now have an enumerated type.\n**apop_opts.mle_trace_path is now the trace_path element of the apop_mle_params struct. Also, it works much better.\n--apop_histogram_normalize function \n--improvements to apop_kernel_density and apop_histogram_print\n\n\tAugust 2007\n--removed apop_model_template. Just copy one of the existing models. \n--apop_data_ptr_ti, apop_data_ptr_ii, apop_data_ptr_it \n--And you can never have too many bug fixes\n\n\tJuly 2007\n--apop_binomial model takes two types of input now: a two-column form with hit count and miss count, and a list of binary hits or misses. \n--apop_lognormal model \n--bug fixes on Information matrix calculation.\n\n\tJune 2007\n[subversion ate this part; sorry.]\n\n\tMay 2007\n--apop_query_to_mixed_data\n**apop_produce_dummies now makes dummy variables from both data and\ntext. This means that there's another parameter you need to set to 'd'\nor 't' to indicate what you want dummified.\n!!**Merged the apop_params and apop_model structures, leaving everything\nin just one struct. That's about all the merging left.\n--apop_text_alloc and apop_text_add to make text manipulation a little\neasier.\n--apop_matrix_apply_all and apop_matrix_map_all operate on all items in\na matrix.\n**small tweak to apop_vector_normalize interface.\n--apop_matrix_inverse and apop_matrix_determinant, because the\napop_det_and_inv interface is sort of ugly.\n--APOP_SUBMATRIX macro\n--MLEs put the expected score in the ->more element of the returned\napop_model. If people find this useful, we can maybe put a\nproper expected_score element in the model.\n--More consts in function headers. You can decide whether this\nis actually useful.\n\n\t0.19 April 2007\n!!**Eliminated the apop_estimate and apop_ep structures, replacing them\nwith the apop_params structure. The apop_params + apop_model pair form a\nclosure representing a parametrized model. Expect the uses element to go\naway soon; after that, things should be stable. Parameters for individual\nmethods now have their own space; try apop_ml_params_alloc and\napop_OLS_params_alloc, for example.\nIf you are just doing things like\napop_estimate_show(apop_OLS.estimate(data,NULL));\nthen don't worry, but if you are doing a lot with the input parameters,\nthen have a look at Chapter 6 of _Modeling with Data_.\n--apop_histogram model\n**Gradually rewriting the histogram functions from before to make use of\nthat model. E.g., eliminated apop_vector_to_cmf.\n**apop_line_to_data fixed to use both vector and matrix terms. Now\nrequires arguments: (indata, vsize, m1size, m2size).\n--MLE now approximates the Information matrix using data gathered during\nthe MLE search. This is wrong but cheap; right but expensive procedures\nforthcoming. [Hint: Simulated annealing gathers more info.]\n--apop_(data|matrix|vector)_fill functions, which are a touch fragile,\nbut very useful when used with care.\n**apop_data_(get|set)_(tn|nt) changed to (ti|it), because n could stand\nfor name or number, while i stands for index, and is often used for integers.\n--apop_names now have a title element, so you can give your data\nstructures a title.\n**apop_params_alloc takes an apop_model, not an &apop_model. It's more\nnatural that way.\n**Finally erradicated every last vestige of inventories: apop_params no\nlonger has a .uses element. Instead, apop_specific_model_params may have\na want_cov, want_expected_value, want_whatever element if the element is\noptional. And really, the parameters themselves should never be\noptional. What was I thinking.\n**apop_model_fix_params now sets up and returns an apop_mle_params object,\nthus resolving the problem that the MLE params needs a model input,\nand the model_fix_params model needed an MLE params input.\n\n\t0.18 March 2007\n**apop_text_to_db now assumes column names unless you specify -nc.\n--If you set parameter_ct==0 in your model definition, the MLEs will\nassign parameter_ct == the number of columns in your data set.\n--Missing data functions: apop_listwise_delete and apop_ml_imputation\n**The constraint element of the apop_model now takes a void* parameter,\nlike it should have all this time.\n**apop_jackknife (1) renamed to apop_jackknife_cov and (2) now actually\nworks.\n**Entirely eliminated the apop_inventory structure. Its sole utility is\ninside the apop_ep struct.\n**Changed the RNG interface for the sake of allowing multidimensional\ndraws. [Not that I have any functions that do that right now.]\n--Bayes-oriented MCMC algorithm: apop_mcmc_update\n!!Bayesian model generator: apop_update\n**apop_model paramters is now an apop_model. See the documentation of\nthe model for all the changes.\n**apop_data_alloc now takes three arguments: vsize, msize1, msize2. To\nupdate, just put a 0, at the head of the arg list.\n!!An absolutely fabulous apop_linear_constraint function.\n--Produce a model with some parameters fixed via apop_model_fix_params.\n--apop_beta model\n\n\tFebruary 2007\n\n**Apop_sv_decomposition has a slightly nicer interface.\n**data->categories was too much to type, and too specific. The apop_data\nstruct now has a data->text element, and textsize[0] and [1]. The\ncategories element is linked to this, but is now deprecated.\n\n\tJanuary 2007\n**Root finding hooked into the max likelihood fns.\n\n0.17\tDecember 2006\n**Apop_model struct has lost the fdf object, which was annoying, and now\nhas the p function.\n--mySQL support.\n**apop_query_to_chars now returns an apop_data structure, so you don't\nhave to go back and gather column names and dimensions.\n**apop_name_get (and the apop_get_tt family) now use regular expressions\ninstead of SQL's LIKE operator. This is _much_ faster.\n--the apop_distribution model.\n\n\t\tNovember 2006\n--The preeminently useful APOP_COL and APOP_ROW\n--apop_data_calloc\n--apop_vector_(apply|map) debugged.\n**apop_estimation_params is just too darn long; reduced to apop_ep.\n\n\n\t\tOctober 2006\n--apop_text_to_db now reads from STDIN.\n**deleted apop_query_db; use apop_query.\n--Kolmogorov-Smirnov test.\n--apop_t_test returns GSL_NAN when given a one-element vector instead of\nhanging.\n--no more soft links in the tgz file==>may work better on Windows machines.\n--apop_(vector|matrix)_(map|apply) will apply a function to every\nrow of a matrix or every element of a vector. The map functions return a\ngsl_vector.\n--bug fix in apop_test_goodness_of_fit.\n\n\t\tSeptember 2006\n\n**Removed apop command-line server thing. It was interesting, but that's\nthe best that could be said of it.\n--Added functions for weighted data: weighted least squares, weighted moments.\n--apop_vector_percentiles now allows for averaging instead of rounding.\n\n\t\tAugust 2006\n**apop_log_likelihood and friends now demand that data be apop_data*,\nrather than void*. Too many things broke when users gave non-apop_data\ndata.\n--bug fixes\n\n\t\tJuly 2006\n--Many more checks for NULL ==> more robust code and easier debugging.\n--Bug fixes.\n**apop_data_split, and apop_data_stack has been revised to handle the\nidea that a vector is the -1st element of the matrix. I.e., check your\ncode if you're trying to merge matrices without merging the vectors.\n--Lattice plots.\n--Convenience t-tests from inside model estimations are fixed.\n--apop_query_to_vector. \n--apop_opts.output_type == 'p' to print to apop_opts.output_pipe\n**apop_..._print and apop_..._show now work out whether elements are\nintegers (if (val == (int) val)...), and print accordingly. This means\nthat apop_..._print_int and apop_..._show_int are basically obsolete,\nand have been removed.\n--Apop_OLS now allows weights.\n--Test library now includes a few NIST certified tests.\n\n\t\tJune 2006\n--added preprocessor cruft to let the library work for C++\n--Jackknife revised\n\n\t\tMay 2006\n**The apop_model no longer includes an inventory. I leave it to the\nestimate function to do its own allocation.\n\n\t\tApril 2006\n**apop_matrix_normalize and apop_vector_normalize had different\nnumbers for the same normalizations. Was that ever dumb. Also, I've\nswitched to chars instead of ints to signify this stuff, for better\nmnemonics without resorting to the\nAPOP_ENUM_YOU_HAVE_TO_LOOK_UP_EVERY_TIME_BECAUSE_ITS_SO_LONG sort of\nthing. If you were using apop_matrix_normalize(data, 0) before, you\nneed to change that to using apop_matrix_normalize(data, 'm'). Thus,\napop_vector_normalize now has one more normalization, for a total of\nfour for both.\n\n--Added apop_rng_alloc convenience fn.\n\n--Added apop_strip_dots to keep inputs to the database healthy.\n\n--apop_name_find uses LIKE instead of strcmp.\n\n--a fn to calculate the generalized harmonic.\n\n--A whole section on histograms and goodness-of-fit tests.\n\n--apop_data_set fns to go with the apop_data_get fns.\n\n--apop_data now includes a vector type\n\t--apop_estimate.parameters is now an apop_data type.\n\t--apop_estimate.names is thus obsolete.\n\n\t\tMarch 2006\n**apop_inventory is now a subset of apop_estimation_params. Implications:\n\t--added apop_estimation_params_alloc() to ensure that inventory is set right.\n\t--the model.estimate(data, inv, params) method is now model.estimate(data, params)\n\t model.estimate(data, NULL) still does what the user expects it to.\n This makes structural sense, but will lightly break any existing code.\n fix: change \n apop_inventory *inv = apop_inventory_set(1);\n model.estimate(data, inv, NULL);\n\n to\n apop_estimation_params *ep = apop_estimation_params_alloc();\n model.estimate(data, ep);\n\n and in any apop_estimates, change any use of est->uses to\n est.estimation_params.uses.\n\n**Next apop_estimate reform: y_values and residuals combined into one\napop_data table with actual, predicted, residual columns.\n\t--obviated the need for a 'dependent' element in apop_names; removed that.\n\tIf you need the name, it's now your_est->dependent->names->colnames[0].\n\n**your_estimate->covariance is now an apop_data set instead of a gsl_matrix.\n\n**the data element of the apop_matrix structure is now named matrix. So\ninstead of data_set->data, use data_set->matrix, and instead of\nestimate->data->data->data, you can use estimate->data->matrix->data.\n\n--The command-line utility has been revisited, and can do a few more\nthings, like OLS.\n\n--Simulated annealing\n\t--added convenience fns apop_vector_distance(pt1,pt2) and\n\t\tapop_vector_grid_distance(pt1,pt2)\n\n**Apop_data_memcpy no longer malloc()s for you, for comparability with\nthe world's other memcpy fns. If you want mallocing, use apop_data_copy.\n\n--apop_test_fisher_exact(). Cut 'n' pasted from R, who cut 'n' pasted it\nfrom somebody else. Despite being the same code, it runs fifty (50)\ntimes faster from Apophenia.\n\n\t\tFebruary 2006\n--sort-of-adaptive MLE: use apop_estimate_restart to execute a new MLE\nsearch beginning where the last one ended, perhaps using a new method or\nrescaled parameters.\n\t--This needed convenience functions to check for divergence, thus\n\t\tadded apop_vector_finite, apop_vector_bounded, apop_vector_isnan.\n**apop_db_to_crosstab now returns an apop_data set instead of a gsl_matrix.\nAlso, it finally works with column headers that aren't numeric.\n**stats like apop_mean are now apop_vector_mean, following the proper\n\tpkg-noun-verb naming scheme. \n--Textbook is much improved.\n--apop_vector_to_pdf convenience fn.\n\n--Some of the fns that used to be of the form\n\tapop_get_something(input, &output);\nare now of the more natural\n\tout\t= apop_get_something(input);\nThis includes apop_array_to_vector and apop_array_to_matrix\n\n--bootstrapping works, and works with with apop_models.\n--apop_poisson model\n\n0.15\tJanuary 2006\nAdded an apop_opts structure for options. Alowed the following changes:\n\t--apop_verbose is now apop_opts.verbose. Try this on your existing code:\n\t\tperl -pi.bak -e 's/apop_verbose/apop_opts.verbose/g' *.c *.h\n\t--the output functions now output into three formats: on screen, to file, to db;\n\t\tsee chapter five of the manual.\n\t\t\n--F tests \n--R squared.\n\n0.14\tDecember 2005\n\nThe apop_data structure, which is just a shell for a gsl_matrix and an\napop_name. Was just sick of sending names following around my tables. \nLets us keep both numerical and categorical data in one place; kind of\nlike R's data frame.\n\n--Added linear model objects: OLS, GLS. This means that what had been\nthe apop_OLS function is now the apop_estimate_OLS function, and where it\nused to take in a gsl_matrix and an apop_name, now it takes an apop_data\nstructure and a NULL. So you'll have to modify your code accordingly.\n\n--A function to generate dummy variables, useful in conjunction with\n--Functions to stack matrices and apop_data sets. Even a\napop_partitioned_OLS function, that will only practically work for small data sets.\n\n--pow(.,.) in the database. I can't believe I dealt with SQL this long w/o it.\n\n0.13\tDecember 2005\n\n--The apop_model object. This was a big deal that deserves more than\njust one line; see the manuals.\n\n0.12 \tmid November 2005\n\n--Bar charts (assuming you've got Gnuplot > 4.1)\n--percentiles (in case you haven't got it)\n**redid MLE system so you can pick among the many options now available. As a part of this:\n\t--better handling of constraints.\n\t--numerical gradients.\n\t--numerical Hessians.\n\n0.12 \tearly November 2005, post-hiatus\n\n--You now have three maximum likelihood estimators to choose from: the\nGSL's no-gradient, the GSL's with-gradient, and Mr. WN's autocalculated\ngradient. \n\n--If you haven't seen it before, the apop_distribution structure is\nincreasingly well-supported. It allows the user to specify the features\nof the Max. Likeihood model in a consistent manner which facilitates\nthings like comparing two models.\n\n--I'd still suggest taking the Waring and Yule distributions with care;\neverything else seems to check out.\n\n\n0.12 \tSeptember 2005\n**The distributions are now objects, which just provides a neat way\nof grouping together the half-dozen functions which are associated with\nany one distribution.\n\n0.11\tSeptember 2005\n--command-line server is much improved. I actually do work with it.\n--Documentation is now via doxygen.\n--asst bug fixes.\n--Have started to take plotting (via gnuplot) seriously\n--a limited test suite. Try: make test .\n\n0.10\tAugust 2005\n--This version includes a server to park itself in memory and receive data\nprocessing requests. The intent is that one can then do analysis from\nthe command line or a Perl/Python/Whatever script. The client/server\nworks in the sense of handling a handful of requests without\nsegfaulting, but remains in proof-of-concept stage. \n--Added apop_merge_db for joining databases, both via C and command line \n--Run t tests from the cmd line or the database.\n\n0.09\tJuly 2005\n--Flattened the relatively complex vasprintf subsystem from GNU, so if\nyou've been having trouble compiling on non-GNU systems, try again.\nAdded two little command-line programs. Also, added more little\nfunctions which aren't very interesting, like t-tests; maybe you'll\nstumble upon them.\n\n0.08\tMay 2005\n--OLS/GLS/MLE now properly support the apop_estimate structure \n--Column names\n\n0.07\tApril 2005\n--uses the apop_estimate structure to return heaps of data from regressions & MLEs\n--uses the apop_model structure\n\n0.06\n--var(x), skew(x), kurtosis(x) added to SQL understood by Apophenia.\n\n0.05\n--added a little crosstab utility\n--queries now accept printf-type arguments. \n\t==>GNU vasprintf was added.\n\t\t==>updated to work with autoconf 1.7\n" }, { "path": "README", "content": "Apophenia is an open statistical library for working with data sets and statistical or simulation models. It provides functions on the same level as those of the typical stats package (such as OLS, probit, or singular value decomposition) but gives the user more flexibility to be creative in model-building. Being in C, it is often an order of magnitude faster when searching for optima or running MCMC chains. The core functions are written in C, but experience has shown them to be easy to bind to Python/Julia/Perl/Ruby/&c.\n\nhttp://apophenia.info/gentle.html provides an overview of the basics of using the library. If you want to know more about the package, see the web site, http://apophenia.info, or have a look at the textbook from Princeton University Press that coevolved with Apophenia, downloadable from http://modelingwithdata.org .\n\n\nThe quick summary for installation:\n\n∙ The library depends on the GNU Scientific Library and SQLite3. If you are using a system with a package manager of some sort, there is certainly a package for them. Be sure to include both the main package and the lib-, -dev, or -devel package. Sample package manager calls:\n\n sudo apt-get install make gcc libgsl0-dev libsqlite3-dev \nor \n sudo yum install make gcc gsl-devel libsqlite3x-devel\nor \n sudo pacman -S make gcc gsl sqlite \n\n∙ The prebuilt package, that has only basic prerequisites (no Autotools or m4) can be downloaded from another Git branch:\n\n #Download the zip file, via wget or your preferred downloading method:\n wget https://github.com/b-k/Apophenia/archive/pkg.zip\n\n #unzip and build\n unzip pkg.zip\n cd Apophenia-pkg\n ./configure\n make\n sudo make install\n\nOr check out the branch via git:\n\n git clone https://github.com/b-k/Apophenia.git\n cd Apophenia\n git checkout pkg\n ./configure\n make\n sudo make install\n\n∙ This master branch of the git repository requires Autotools, so it can build the\npackage. Try (apt-get || yum install) autoconf automake libtool. If you have Autotools installed, then from this branch you can run:\n\n ./configure\n cd apophenia-1.0\n make \n sudo make install\n\n∙ Find detailed setup instructions and some troubleshooting notes at\nhttp://apophenia.info/setup.html .\n\n\nThanks for your interest. I do hope that Apophenia helps you learn more from your data.\n\n--BK\n\nPS: Lawyers, please note that a file named COPYING in the install/ directory describes how this package is licensed under GPLv2.\n" }, { "path": "apop.m4.h", "content": "/** \\file */\n/* Copyright (c) 2005--2014 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n/* Here are the headers for all of apophenia's functions, typedefs, static variables and\nmacros. All of these begin with the apop_ (or Apop_ or APOP_) prefix.\n\nThere used to be a series of sub-headers, but they never provided any serious\nbenefit. Please use your text editor's word-search feature to find any elements you\nmay be looking for. About a third of the file is comments and doxygen documentation,\nso syntax highlighting that distinguishes code from comments will also help to make\nthis more navigable.*/\n\n/** \\defgroup all_public Public functions, structs, and types\n\\addtogroup all_public\n@{\n*/\n\n#pragma once\n#ifdef\t__cplusplus\nextern \"C\" {\n#endif\n\n/** \\cond doxy_ignore */\n#ifndef _GNU_SOURCE\n#define _GNU_SOURCE //for asprintf\n#endif\n\n#include \n#include //raise(SIGTRAP)\n#include \n#include \n#include \n\n\n //////Optional arguments\n\n/* A means of providing more script-like means of sending arguments to a function.\n\nThese macros are intended as internal. If you are interested in using this mechanism\nin out-of-Apophenia work, grep docs/documentation.h for optionaldetails to find notes\non how these are used (Doxygen doesn't use that page),\n*/\n#define apop_varad_head(type, name) type variadic_##name(variadic_type_##name varad_in)\n\n#define apop_varad_declare(type, name, ...) \\\n typedef struct { \\\n __VA_ARGS__ ; \\\n } variadic_type_##name; \\\n apop_varad_head(type, name);\n\n#define apop_varad_var(name, value) name = varad_in.name ? varad_in.name : (value);\n#define apop_varad_link(name,...) variadic_##name((variadic_type_##name) {__VA_ARGS__})\n\n/** \\endcond */ //End of Doxygen ignore.\n\n\n //////The types and functions that act on them\n\n/** This structure holds the names of the components of the \\ref apop_data set. You may never have to worry about it directly, because most operations on \\ref apop_data sets will take care of the names for you.\n*/\ntypedef struct{\n char *title;\n\tchar * vector;\n\tchar ** col;\n\tchar ** row;\n\tchar ** text;\n\tint colct, rowct, textct;\n unsigned long *colhash, *rowhash, *texthash;\n} apop_name;\n\n/** The \\ref apop_data structure represents a data set. See \\ref dataoverview.*/\ntypedef struct apop_data{\n gsl_vector *vector;\n gsl_matrix *matrix;\n apop_name *names;\n char ***text;\n size_t textsize[2];\n gsl_vector *weights;\n struct apop_data *more;\n char error;\n} apop_data;\n\n/* Settings groups. For internal use only; see apop_settings.c and \n settings.h for related machinery. */\ntypedef struct {\n char name[101];\n unsigned long name_hash;\n void *setting_group;\n void *copy;\n void *free;\n} apop_settings_type;\n\n/** A statistical model. See \\ref modelsec for details. */\ntypedef struct apop_model apop_model;\n\n/** The elements of the \\ref apop_model type, representing a statistical model. See \\ref\n modelsec and \\ref modeldetails for use and details. */\nstruct apop_model{\n char name[101]; \n int vsize, msize1, msize2, dsize;\n apop_data *data;\n apop_data *parameters;\n apop_data *info;\n void (*estimate)(apop_data * data, apop_model *params); \n long double (*p)(apop_data *d, apop_model *params);\n long double (*log_likelihood)(apop_data *d, apop_model *params);\n long double (*cdf)(apop_data *d, apop_model *params);\n long double (*constraint)(apop_data *data, apop_model *params);\n int (*draw)(double *out, gsl_rng* r, apop_model *params);\n void (*prep)(apop_data *data, apop_model *params);\n apop_settings_type *settings;\n void *more;\n size_t more_size;\n char error;\n};\n\n/** The global options. */\ntypedef struct{\n int verbose; /**< Set this to zero for silent mode, one for errors and warnings. default = 0. */\n char stop_on_warning; /**< See \\ref debugging . */\n char output_delimiter[100]; /**< The separator between elements of output tables. The default is \"\\t\", but \n for LaTeX, use \"&\\t\", or use \"|\" to get pipe-delimited output. */\n char input_delimiters[100]; /**< Deprecated. Please use per-function inputs to \\ref apop_text_to_db and \\ref apop_text_to_data. Default = \"|,\\t\" */\n char *db_name_column; /**< If not NULL or \"\", the name of the column in your tables that holds row names.*/\n char *nan_string; /**< The string used to indicate NaN. Default: \"NaN. Comparisons are case-insensitive.*/\n char db_engine; /**< If this is 'm', use mySQL, else use SQLite. */\n char db_user[101]; /**< Username for database login. Max 100 chars. */\n char db_pass[101]; /**< Password for database login. Max 100 chars. */\n FILE *log_file; /**< The file handle for the log. Defaults to \\c stderr, but change it with, e.g.,\n apop_opts.log_file = fopen(\"outlog\", \"w\"); */\n\n#define Autoconf_no_atomics @Autoconf_no_atomics@\n\n #if __STDC_VERSION__ > 201100L && !defined(__STDC_NO_ATOMICS__) && Autoconf_no_atomics==0\n _Atomic(int) rng_seed;\n #else\n int rng_seed;\n #endif\n float version;\n} apop_opts_type;\n\nextern apop_opts_type apop_opts;\n\napop_name * apop_name_alloc(void);\nint apop_name_add(apop_name * n, char const *add_me, char type);\nvoid apop_name_free(apop_name * free_me);\nvoid apop_name_print(apop_name * n);\nApop_var_declare( void apop_name_stack(apop_name * n1, apop_name *nadd, char type1, char typeadd) )\napop_name * apop_name_copy(apop_name *in);\nint apop_name_find(const apop_name *n, const char *findme, const char type);\n\nvoid apop_data_add_names_base(apop_data *d, const char type, char const ** names);\n\n/** Add a list of names to a data set.\n\n\\li Use this with a list of names that you type in yourself, like\n\\code\napop_data_add_names(mydata, 'c', \"age\", \"sex\", \"height\");\n\\endcode\nNotice the lack of curly braces around the list.\n\n\\li You may have an array of names, probably autogenerated, that you would like to\nadd. In this case, make certain that the last element of the array is \\c NULL, and\ncall the base function:\n\\code\nchar **[] colnames = {\"age\", \"sex\", \"height\", NULL};\napop_data_add_names_base(mydata, 'c', colnames);\n\\endcode\nBut if you forget the \\c NULL marker, this has good odds of segfaulting. You may prefer to use a \\c for loop that inserts each name in turn using \\ref apop_name_add.\n\n\\see \\ref apop_name_add, although \\ref apop_data_add_names will be more useful in most cases. \n*/\n#define apop_data_add_names(dataset, type, ...) apop_data_add_names_base((dataset), (type), (char const*[]) {__VA_ARGS__, NULL}) \n\n\n/** Free an \\ref apop_data structure.\n \n\\li As with \\c free(), it is safe to send in a \\c NULL pointer (in which case the function does nothing).\n\\li If the \\c more pointer is not \\c NULL, I will free the pointed-to data set first.\nIf you don't want to free data sets down the chain, set more=NULL before calling this.\n\\li This is actually a macro (that calls \\ref apop_data_free_base). It\nsets \\c freeme to \\c NULL when it's done, because there's nothing safe you can do with the\nfreed location, and you can later safely test conditions like if (data) ....\n*/\n#define apop_data_free(freeme) (apop_data_free_base(freeme) ? 0 : ((freeme)= NULL))\n\nchar apop_data_free_base(apop_data *freeme);\nApop_var_declare( apop_data * apop_data_alloc(const size_t size1, const size_t size2, const int size3) )\nApop_var_declare( apop_data * apop_data_calloc(const size_t size1, const size_t size2, const int size3) )\nApop_var_declare( apop_data * apop_data_stack(apop_data *m1, apop_data * m2, char posn, char inplace) )\napop_data ** apop_data_split(apop_data *in, int splitpoint, char r_or_c);\napop_data * apop_data_copy(const apop_data *in);\nvoid apop_data_rm_columns(apop_data *d, int *drop);\nvoid apop_data_memcpy(apop_data *out, const apop_data *in);\nApop_var_declare( double * apop_data_ptr(apop_data *data, int row, int col, const char *rowname, const char *colname, const char *page) )\nApop_var_declare( double apop_data_get(const apop_data *data, size_t row, int col, const char *rowname, const char *colname, const char *page) )\nApop_var_declare( int apop_data_set(apop_data *data, size_t row, int col, const double val, const char *rowname, const char * colname, const char *page) )\nvoid apop_data_add_named_elmt(apop_data *d, char *name, double val);\nint apop_text_set(apop_data *in, const size_t row, const size_t col, const char *fmt, ...);\napop_data * apop_text_alloc(apop_data *in, const size_t row, const size_t col);\nvoid apop_text_free(char ***freeme, int rows, int cols);\nApop_var_declare( apop_data * apop_data_transpose(apop_data *in, char transpose_text, char inplace) )\ngsl_matrix * apop_matrix_realloc(gsl_matrix *m, size_t newheight, size_t newwidth);\ngsl_vector * apop_vector_realloc(gsl_vector *v, size_t newheight);\n\n#define apop_data_prune_columns(in, ...) apop_data_prune_columns_base((in), (char *[]) {__VA_ARGS__, NULL})\napop_data* apop_data_prune_columns_base(apop_data *d, char **colnames);\n\nApop_var_declare( apop_data * apop_data_get_page(const apop_data * data, const char * title, const char match) )\napop_data * apop_data_add_page(apop_data * dataset, apop_data *newpage,const char *title);\nApop_var_declare( apop_data* apop_data_rm_page(apop_data * data, const char *title, const char free_p) )\nApop_var_declare( apop_data * apop_data_rm_rows(apop_data *in, int *drop, int (*do_drop)(apop_data* ! void*), void* drop_parameter) )\n\n//in apop_asst.c:\nApop_var_declare( apop_data * apop_model_draws(apop_model *model, int count, apop_data *draws) )\n\n\n/* Convenience functions to convert among vectors (gsl_vector), matrices (gsl_matrix), \n arrays (double **), and database tables */\n\n//From vector\ngsl_vector *apop_vector_copy(const gsl_vector *in);\nApop_var_declare( gsl_matrix * apop_vector_to_matrix(const gsl_vector *in, char row_col) )\n\n//From matrix\ngsl_matrix *apop_matrix_copy(const gsl_matrix *in);\nApop_var_declare( apop_data *apop_db_to_crosstab(char const*tabname, char const*row, char const*col, char const*data, char is_aggregate) )\n\n//From array\nApop_var_declare( gsl_vector * apop_array_to_vector(double *in, int size) )\n/** \\cond doxy_ignore */ //Deprecated\n#define apop_text_add apop_text_set\n#define apop_line_to_vector apop_array_to_vector\n/** \\endcond */\n\n//From text\nApop_var_declare( apop_data * apop_text_to_data(char const *text_file, int has_row_names, int has_col_names, int const *field_ends, char const *delimiters) )\nApop_var_declare( int apop_text_to_db(char const *text_file, char *tabname, int has_row_names, int has_col_names, char **field_names, int const *field_ends, apop_data *field_params, char *table_params, char const *delimiters, char if_table_exists) )\n\n//rank data\napop_data *apop_data_rank_expand (apop_data *in);\nApop_var_declare( apop_data *apop_data_rank_compress (apop_data *in, int min_bins) )\n\n//From crosstabs\nvoid apop_crosstab_to_db(apop_data *in, char *tabname, char *row_col_name, \n\t\t\t\t\t\tchar *col_col_name, char *data_col_name);\n\n//packing data into a vector\nApop_var_declare( gsl_vector * apop_data_pack(const apop_data *in, gsl_vector *out, char more_pages, char use_info_pages) )\nApop_var_declare( void apop_data_unpack(const gsl_vector *in, apop_data *d, char use_info_pages) )\n\n#define apop_vector_fill(avfin, ...) apop_vector_fill_base((avfin), (double []) {__VA_ARGS__})\n#define apop_data_fill(adfin, ...) apop_data_fill_base((adfin), (double []) {__VA_ARGS__})\n#define apop_text_fill(dataset, ...) apop_text_fill_base((dataset), (char* []) {__VA_ARGS__, NULL})\n#define apop_data_falloc(sizes, ...) apop_data_fill(apop_data_alloc sizes, __VA_ARGS__)\n \napop_data *apop_data_fill_base(apop_data *in, double []);\ngsl_vector *apop_vector_fill_base(gsl_vector *in, double []);\napop_data *apop_text_fill_base(apop_data *data, char* text[]);\n\n //// Models and model support functions\n\nextern apop_model *apop_beta;\nextern apop_model *apop_bernoulli;\nextern apop_model *apop_binomial;\nextern apop_model *apop_chi_squared;\nextern apop_model *apop_dirichlet;\nextern apop_model *apop_exponential;\nextern apop_model *apop_f_distribution;\nextern apop_model *apop_gamma;\nextern apop_model *apop_improper_uniform;\nextern apop_model *apop_iv;\nextern apop_model *apop_kernel_density;\nextern apop_model *apop_loess;\nextern apop_model *apop_logit;\nextern apop_model *apop_lognormal;\nextern apop_model *apop_multinomial;\nextern apop_model *apop_multivariate_normal;\nextern apop_model *apop_normal;\nextern apop_model *apop_ols;\nextern apop_model *apop_pmf;\nextern apop_model *apop_poisson;\nextern apop_model *apop_probit;\nextern apop_model *apop_t_distribution;\nextern apop_model *apop_uniform;\n//extern apop_model *apop_wishart;\nextern apop_model *apop_wls;\nextern apop_model *apop_yule;\nextern apop_model *apop_zipf;\n\n//model transformations\nextern apop_model *apop_coordinate_transform;\nextern apop_model *apop_composition;\nextern apop_model *apop_dconstrain;\nextern apop_model *apop_mixture;\nextern apop_model *apop_cross;\n\n/** Alias for the \\ref apop_normal distribution, qv. */\n#define apop_gaussian apop_normal\n#define apop_OLS apop_ols\n#define apop_PMF apop_pmf\n#define apop_F_distribution apop_f_distribution\n#define apop_IV apop_iv\n\n\nvoid apop_model_free (apop_model * free_me);\nApop_var_declare( void apop_model_print (apop_model * model, FILE *output_pipe) )\nvoid apop_model_show (apop_model * print_me); //deprecated\napop_model * apop_model_copy(apop_model *in); //in apop_model.c\napop_model * apop_model_clear(apop_data * data, apop_model *model);\n\napop_model * apop_estimate(apop_data *d, apop_model *m);\nvoid apop_score(apop_data *d, gsl_vector *out, apop_model *m);\ndouble apop_log_likelihood(apop_data *d, apop_model *m);\ndouble apop_p(apop_data *d, apop_model *m);\ndouble apop_cdf(apop_data *d, apop_model *m);\nint apop_draw(double *out, gsl_rng *r, apop_model *m);\nvoid apop_prep(apop_data *d, apop_model *m);\napop_model *apop_parameter_model(apop_data *d, apop_model *m);\napop_data * apop_predict(apop_data *d, apop_model *m);\n\napop_model *apop_beta_from_mean_var(double m, double v); //in apop_beta.c\n\n#define apop_model_set_parameters(in, ...) apop_model_set_parameters_base((in), (double []) {__VA_ARGS__})\napop_model *apop_model_set_parameters_base(apop_model *in, double ap[]);\n\n//apop_mixture.c\n/** Produce a model as a linear combination of other models. See the documentation for the \\ref apop_mixture model. \n\n\\param ... A list of models, either all parameterized or all unparameterized. See\nexamples in the \\ref apop_mixture documentation.\n*/\n#define apop_model_mixture(...) apop_model_mixture_base((apop_model *[]){__VA_ARGS__, NULL})\napop_model *apop_model_mixture_base(apop_model **inlist);\n\n//transform/apop_cross.c.\n\n/** Generate a model consisting of the cross product of several independent models. The output \\ref apop_model\nis a copy of \\ref apop_cross; see that model's documentation for details.\n\n\\li If you input only one model, return a copy of that model; print a warning iff apop_opts.verbose >= 2.\n\n\\exception error=='n' First model input is \\c NULL.\n\nExamples:\n\n\\include cross_models.c\n*/\n#define apop_model_cross(...) apop_model_cross_base((apop_model *[]){__VA_ARGS__, NULL})\napop_model *apop_model_cross_base(apop_model *mlist[]);\n\n ////More functions\n\n //The variadic versions, with lots of options to input extra parameters to the\n //function being mapped/applied\nApop_var_declare( apop_data * apop_map(apop_data *in, double (*fn_d)(double), double (*fn_v)(gsl_vector*),\n double (*fn_r)(apop_data *), double (*fn_dp)(double! void *), double (*fn_vp)(gsl_vector*! void *),\n double (*fn_rp)(apop_data *! void *), double (*fn_dpi)(double! void *! int),\n double (*fn_vpi)(gsl_vector*! void *! int), double (*fn_rpi)(apop_data*! void *! int),\n double (*fn_di)(double! int), double (*fn_vi)(gsl_vector*! int), double (*fn_ri)(apop_data*! int),\n void *param, int inplace, char part, int all_pages) )\nApop_var_declare( double apop_map_sum(apop_data *in, double (*fn_d)(double), double (*fn_v)(gsl_vector*),\n double (*fn_r)(apop_data *), double (*fn_dp)(double! void *), double (*fn_vp)(gsl_vector*! void *),\n double (*fn_rp)(apop_data *! void *), double (*fn_dpi)(double! void *! int),\n double (*fn_vpi)(gsl_vector*! void *! int), double (*fn_rpi)(apop_data*! void *! int),\n double (*fn_di)(double! int), double (*fn_vi)(gsl_vector*! int), double (*fn_ri)(apop_data*! int),\n void *param, char part, int all_pages) )\n\n //the specific-to-a-type versions, quicker and easier when appropriate.\ngsl_vector *apop_matrix_map(const gsl_matrix *m, double (*fn)(gsl_vector*));\ngsl_vector *apop_vector_map(const gsl_vector *v, double (*fn)(double));\nvoid apop_matrix_apply(gsl_matrix *m, void (*fn)(gsl_vector*));\nvoid apop_vector_apply(gsl_vector *v, void (*fn)(double*));\ngsl_matrix * apop_matrix_map_all(const gsl_matrix *in, double (*fn)(double));\nvoid apop_matrix_apply_all(gsl_matrix *in, void (*fn)(double *));\n\ndouble apop_vector_map_sum(const gsl_vector *in, double(*fn)(double));\ndouble apop_matrix_map_sum(const gsl_matrix *in, double (*fn)(gsl_vector*));\ndouble apop_matrix_map_all_sum(const gsl_matrix *in, double (*fn)(double));\n\n\n // Some output routines\nApop_var_declare( void apop_matrix_print(const gsl_matrix *data, char const *output_name, FILE *output_pipe, char output_type, char output_append) )\nApop_var_declare( void apop_vector_print(gsl_vector *data, char const *output_name, FILE *output_pipe, char output_type, char output_append) )\nApop_var_declare( void apop_data_print(const apop_data *data, char const *output_name, FILE *output_pipe, char output_type, char output_append) )\n\nvoid apop_matrix_show(const gsl_matrix *data);\nvoid apop_vector_show(const gsl_vector *data);\nvoid apop_data_show(const apop_data *data);\n\n\n //statistics\nApop_var_declare( double apop_vector_mean(gsl_vector const *v, gsl_vector const *weights))\nApop_var_declare( double apop_vector_var(gsl_vector const *v, gsl_vector const *weights))\nApop_var_declare( double apop_vector_skew_pop(gsl_vector const *v, gsl_vector const *weights))\nApop_var_declare( double apop_vector_kurtosis_pop(gsl_vector const *v, gsl_vector const *weights))\nApop_var_declare( double apop_vector_cov(gsl_vector const *v1, gsl_vector const *v2,\n gsl_vector const *weights))\n\nApop_var_declare( double apop_vector_distance(const gsl_vector *ina, const gsl_vector *inb, const char metric, const double norm) )\n\nApop_var_declare( void apop_vector_normalize(gsl_vector *in, gsl_vector **out, const char normalization_type) )\n\napop_data * apop_data_covariance(const apop_data *in);\napop_data * apop_data_correlation(const apop_data *in);\nlong double apop_vector_entropy(gsl_vector *in);\nlong double apop_matrix_sum(const gsl_matrix *m);\ndouble apop_matrix_mean(const gsl_matrix *data);\nvoid apop_matrix_mean_and_var(const gsl_matrix *data, double *mean, double *var);\napop_data * apop_data_summarize(apop_data *data);\nApop_var_declare( double * apop_vector_percentiles(gsl_vector *data, char rounding) )\n\napop_data *apop_test_fisher_exact(apop_data *intab); //in apop_fisher.c\n\n//from apop_t_f_chi.c:\nApop_var_declare( int apop_matrix_is_positive_semidefinite(gsl_matrix *m, char semi) )\ndouble apop_matrix_to_positive_semidefinite(gsl_matrix *m);\nlong double apop_multivariate_gamma(double a, int p);\nlong double apop_multivariate_lngamma(double a, int p);\n\n//apop_tests.c\napop_data *\tapop_t_test(gsl_vector *a, gsl_vector *b);\napop_data *\tapop_paired_t_test(gsl_vector *a, gsl_vector *b);\nApop_var_declare( apop_data* apop_anova(char *table, char *data, char *grouping1, char *grouping2) )\n#define apop_ANOVA apop_anova\nApop_var_declare( apop_data * apop_f_test (apop_model *est, apop_data *contrast) )\n#define apop_F_test apop_f_test\n\n//from the regression code:\n#define apop_estimate_r_squared(in) apop_estimate_coefficient_of_determination(in)\n\napop_data * apop_text_unique_elements(const apop_data *d, size_t col);\ngsl_vector * apop_vector_unique_elements(const gsl_vector *v);\nApop_var_declare( apop_data * apop_data_to_factors(apop_data *data, char intype, int incol, int outcol) )\nApop_var_declare( apop_data * apop_data_get_factor_names(apop_data *data, int col, char type) )\n\nApop_var_declare( apop_data * apop_data_to_dummies(apop_data *d, int col, char type, int keep_first, char append, char remove) )\n\nApop_var_declare( long double apop_model_entropy(apop_model *in, int draws) )\nApop_var_declare( long double apop_kl_divergence(apop_model *from, apop_model *to, int draw_ct, gsl_rng *rng) )\n\napop_data *apop_estimate_coefficient_of_determination (apop_model *);\nvoid apop_estimate_parameter_tests (apop_model *est);\n\n//Bootstrapping & RNG\napop_data * apop_jackknife_cov(apop_data *data, apop_model *model);\nApop_var_declare( apop_data * apop_bootstrap_cov(apop_data *data, apop_model *model, gsl_rng* rng, int iterations, char keep_boots, char ignore_nans, apop_data **boot_store) )\ngsl_rng *apop_rng_alloc(int seed);\ndouble apop_rng_GHgB3(gsl_rng * r, double* a); //in apop_asst.c\n\n#define apop_rng_get_thread(thread_in) apop_rng_get_thread_base(#thread_in[0]=='\\0' ? -1: (thread_in+0))\ngsl_rng *apop_rng_get_thread_base(int thread);\n\nint apop_arms_draw (double *out, gsl_rng *r, apop_model *m);\n\n\n // maximum likelihod estimation related functions\n\nApop_var_declare( gsl_vector * apop_numerical_gradient(apop_data * data, apop_model* model, double delta) )\nApop_var_declare( apop_data * apop_model_hessian(apop_data * data, apop_model *model, double delta) )\nApop_var_declare( apop_data * apop_model_numerical_covariance(apop_data * data, apop_model *model, double delta) )\n\nvoid apop_maximum_likelihood(apop_data * data, apop_model *dist);\n\nApop_var_declare( apop_model * apop_estimate_restart (apop_model *e, apop_model *copy, char * starting_pt, double boundary) )\n\n//in apop_linear_constraint.c\nApop_var_declare( long double apop_linear_constraint(gsl_vector *beta, apop_data * constraint, double margin) )\n\n//in apop_model_fix_params.c\napop_model * apop_model_fix_params(apop_model *model_in);\napop_model * apop_model_fix_params_get_base(apop_model *model_in);\n\n\n\n //////vtables\n/** \\cond doxy_ignore */\n\n/* This declares the vtable macros for each procedure that uses the mechanism.\n\n--We want to have type-checking on the functions put into the vtables. Type checking\nhappens only with functions, not macros, so we need a type_check function for every\nvtable.\n\n--Only once in your codebase, you'll need to #define Declare_type_checking_fns to\nactually define the type checking function. Everywhere else, the function is merely\ndeclared.\n\n--All other uses point to having a macro, such as using __VA_ARGS__ to allow any sort\nof inputs to the hash.\n\n--We want to have such a macro for every vtable. That means that we need a macro\nto write macros. We can't do that with C macros, so this file uses m4 macros to\ngenerate C macros.\n\n--After the m4 definition of make_vtab_fns, each new vtable requires a typedef, a hash\ndefinition, and a call to make_vtab_fns to do the rest.\n*/\nm4_define(make_vtab_fns, <|m4_dnl\n#ifdef Declare_type_checking_fns\nvoid $1_type_check($1_type in){ };\n#else\nvoid $1_type_check($1_type in);\n#endif\n#define $1_vtable_add(fn, ...) $1_type_check(fn), apop_vtable_add(\"$1\", fn, $1_hash(__VA_ARGS__))\n#define $1_vtable_get(...) apop_vtable_get(\"$1\", $1_hash(__VA_ARGS__))\n#define $1_vtable_drop(...) apop_vtable_drop(\"$1\", $1_hash(__VA_ARGS__))m4_dnl\n|>)\n\nint apop_vtable_add(char const *tabname, void *fn_in, unsigned long hash);\nvoid *apop_vtable_get(char const *tabname, unsigned long hash);\nint apop_vtable_drop(char const *tabname, unsigned long hash);\n\ntypedef apop_model *(*apop_update_type)(apop_data *, apop_model* , apop_model*);\n#define apop_update_hash(m1, m2) ( \\\n ((m1)->log_likelihood ? (size_t)(m1)->log_likelihood : \\\n (m1)->p ? (size_t)(m1)->p*33 : \\\n (m1)->draw ? (size_t)(m1)->draw*33*27 \\\n : 33*27*19) \\\n +((m2)->log_likelihood ? (size_t)(m2)->log_likelihood : \\\n (m2)->p ? (size_t)(m2)->p*33 : \\\n (m2)->draw ? (size_t)(m2)->draw*33*27 \\\n : 33*27*19 \\\n ) * 37)\nmake_vtab_fns(apop_update)\n\ntypedef long double (*apop_entropy_type)(apop_model *model);\n#define apop_entropy_hash(m1) ((size_t)(m1)->log_likelihood + 33 * (size_t)((m1)->p) + 27*(size_t)((m1)->draw))\nmake_vtab_fns(apop_entropy)\n\ntypedef void (*apop_score_type)(apop_data *d, gsl_vector *gradient, apop_model *params);\n#define apop_score_hash(m1) ((size_t)((m1)->log_likelihood ? (m1)->log_likelihood : (m1)->p))\nmake_vtab_fns(apop_score)\n\ntypedef apop_model* (*apop_parameter_model_type)(apop_data *, apop_model *);\n#define apop_parameter_model_hash(m1) ((size_t)((m1)->log_likelihood ? (m1)->log_likelihood : (m1)->p)*33 + (m1)->estimate ? (size_t)(m1)->estimate: 27)\nmake_vtab_fns(apop_parameter_model)\n\ntypedef apop_data * (*apop_predict_type)(apop_data *d, apop_model *params);\n#define apop_predict_hash(m1) ((size_t)((m1)->log_likelihood ? (m1)->log_likelihood : (m1)->p)*33 + (m1)->estimate ? (size_t)(m1)->estimate: 27)\nmake_vtab_fns(apop_predict)\n\ntypedef void (*apop_model_print_type)(apop_model *params, FILE *out);\n#define apop_model_print_hash(m1) ((m1)->log_likelihood ? (size_t)(m1)->log_likelihood : \\\n (m1)->p ? (size_t)(m1)->p*33 : \\\n (m1)->estimate ? (size_t)(m1)->estimate*33*33 : \\\n (m1)->draw ? (size_t)(m1)->draw*33*27 : \\\n (m1)->cdf ? (size_t)(m1)->cdf*27*27 \\\n : 27)\nmake_vtab_fns(apop_model_print)\n\n/** \\endcond */ //End of Doxygen ignore.\n\n\n //////Asst\n\nlong double apop_generalized_harmonic(int N, double s) __attribute__ ((__pure__));\n\napop_data * apop_test_anova_independence(apop_data *d);\n#define apop_test_ANOVA_independence(d) apop_test_anova_independence(d)\n\nApop_var_declare( int apop_regex(const char *string, const char* regex, apop_data **substrings, const char use_case) )\n\nint apop_system(const char *fmt, ...) __attribute__ ((format (printf,1,2)));\n\n//Histograms and PMFs\ngsl_vector * apop_vector_moving_average(gsl_vector *, size_t);\napop_data * apop_histograms_test_goodness_of_fit(apop_model *h0, apop_model *h1);\napop_data * apop_test_kolmogorov(apop_model *m1, apop_model *m2);\napop_data *apop_data_pmf_compress(apop_data *in);\nApop_var_declare( apop_data * apop_data_to_bins(apop_data const *indata, apop_data const *binspec, int bin_count, char close_top_bin) )\nApop_var_declare( apop_model * apop_model_to_pmf(apop_model *model, apop_data *binspec, long int draws, int bin_count) )\n\n//text conveniences\nApop_var_declare( char* apop_text_paste(apop_data const*strings, char *between, char *before, char *after, char *between_cols, int (*prune)(apop_data* ! int ! int ! void*), void* prune_parameter) )\n/** Notify the user of errors, warning, or debug info. \n\nwrites to \\ref apop_opts.log_file, which is a \\c FILE handle. The default is \\c stderr,\nbut use \\c fopen to attach to a file.\n\n \\param verbosity At what verbosity level should the user be warned? E.g., if level==2, then print iff apop_opts.verbosity >= 2.\n \\param ... The message to write to the log (presuming the verbosity level is high\nenough). This can be a printf-style format with following arguments, \ne.g., apop_notify(0, \"Beta is currently %g\", beta).\n*/\n#define Apop_notify(verbosity, ...) {\\\n if (apop_opts.verbose != -1 && apop_opts.verbose >= verbosity) { \\\n if (!apop_opts.log_file) apop_opts.log_file = stderr; \\\n fprintf(apop_opts.log_file, \"%s: \", __func__); fprintf(apop_opts.log_file, __VA_ARGS__); fprintf(apop_opts.log_file, \"\\n\"); \\\n fflush(apop_opts.log_file); \\\n} }\n\n/** \\cond doxy_ignore */\n#define Apop_maybe_abort(level) \\\n {if ((apop_opts.verbose >= level && apop_opts.stop_on_warning == 'v') \\\n || (apop_opts.stop_on_warning=='w') ) \\\n raise(SIGTRAP);}\n/** \\endcond */\n\n/** Execute an action and print a message to the current \\c FILE handle held by apop_opts.log_file (default: \\c stderr).\n \n\\param test The expression that, if true, triggers the action.\n\\param onfail If the assertion fails, do this. E.g., out->error='x'; return GSL_NAN. Notice that it is OK to include several lines of semicolon-separated code here, but if you have a lot to do, the most readable option may be goto outro, plus an appropriately-labeled section at the end of your function.\n\\param level Print the warning message only if \\ref apop_opts_type \"apop_opts.verbose\" is greater than or equal to this. Zero usually works, but for minor infractions use one, or for more verbose debugging output use 2.\n\\param ... The error message in printf form, plus any arguments to be inserted into the printf string. I'll provide the function name and a carriage return.\n\nSome examples:\n\n\\code\n//the typical case, stopping function execution:\nApop_stopif(isnan(x), return NAN, 0, \"x is NAN; failing\");\n\n//Mark a flag, go to a cleanup step\nApop_stopif(x < 0, needs_cleanup=1; goto cleanup, 0, \"x is %g; cleaning up and exiting.\", x);\n\n//Print a diagnostic iff apop_opts.verbose>=1 and continue\nApop_stopif(x < 0, , 1, \"warning: x is %g.\", x);\n\\endcode\n\n\\li If \\c apop_opts.stop_on_warning is nonzero and not 'v', then a failed test halts via \\c abort(), even if the apop_opts.verbose level is set so that the warning message doesn't print to screen. Use this when running via debugger.\n\\li If \\c apop_opts.stop_on_warning is 'v', then a failed test halts via \\c abort() iff the verbosity level is high enough to print the error.\n*/\n#define Apop_stopif(test, onfail, level, ...) do {\\\n if (test) { \\\n Apop_notify(level, __VA_ARGS__); \\\n Apop_maybe_abort(level) \\\n onfail; \\\n } } while(0)\n\n#define apop_errorlevel -5\n\n/** \\cond doxy_ignore */\n//For use in stopif, to return a blank apop_data set with an error attached.\n#define apop_return_data_error(E) {apop_data *out=apop_data_alloc(); out->error='E'; return out;}\n\n/* The Apop_stopif macro is currently favored, but there's a long history of prior\n error-handling setups. Consider all of the Assert... macros below to be deprecated.\n*/\n#define Apop_assert_c(test, returnval, level, ...) \\\n Apop_stopif(!(test), return returnval, level, __VA_ARGS__)\n\n#define Apop_assert(test, ...) Apop_assert_c((test), 0, apop_errorlevel, __VA_ARGS__)\n\n//For things that return void. Transitional and deprecated at birth.\n#define Apop_assert_n(test, ...) Apop_assert_c((test), , apop_errorlevel, __VA_ARGS__)\n#define Apop_assert_negone(test, ...) Apop_assert_c((test), -1, apop_errorlevel, __VA_ARGS__)\n/** \\endcond */ //End of Doxygen ignore.\n\n//Missing data\nApop_var_declare( apop_data * apop_data_listwise_delete(apop_data *d, char inplace) )\napop_model * apop_ml_impute(apop_data *d, apop_model* meanvar);\n\nApop_var_declare(apop_model *apop_model_metropolis(apop_data *d, gsl_rng* rng, apop_model *m))\nApop_var_declare( apop_model * apop_update(apop_data *data, apop_model *prior, apop_model *likelihood, gsl_rng *rng) )\n\nApop_var_declare( double apop_test(double statistic, char *distribution, double p1, double p2, char tail) )\n\n//apop_sort.c\nApop_var_declare( apop_data *apop_data_sort(apop_data *data, apop_data *sort_order, char asc, char inplace, double *col_order))\n\n//raking\nApop_var_declare( apop_data * apop_rake(char const *margin_table, char * const*var_list, \n int var_ct, char * const *contrasts, int contrast_ct, \n char const *structural_zeros, int max_iterations, double tolerance, \n char const *count_col, char const *init_table, \n char const *init_count_col, double nudge) )\n\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n\n //Some linear algebra utilities\n\ndouble apop_det_and_inv(const gsl_matrix *in, gsl_matrix **out, int calc_det, int calc_inv);\nApop_var_declare( apop_data * apop_dot(const apop_data *d1, const apop_data *d2, char form1, char form2) )\nApop_var_declare( int apop_vector_bounded(const gsl_vector *in, long double max) )\ngsl_matrix * apop_matrix_inverse(const gsl_matrix *in) ;\ndouble apop_matrix_determinant(const gsl_matrix *in) ;\n//apop_data* apop_sv_decomposition(gsl_matrix *data, int dimensions_we_want);\nApop_var_declare( apop_data * apop_matrix_pca(gsl_matrix *data, int const dimensions_we_want) )\nApop_var_declare( gsl_vector * apop_vector_stack(gsl_vector *v1, gsl_vector const * v2, char inplace) )\nApop_var_declare( gsl_matrix * apop_matrix_stack(gsl_matrix *m1, gsl_matrix const * m2, char posn, char inplace) )\n\nvoid apop_vector_log(gsl_vector *v);\nvoid apop_vector_log10(gsl_vector *v);\nvoid apop_vector_exp(gsl_vector *v);\n\n ////Subsetting macros\n\n/** \\cond doxy_ignore */\n/** These are all deprecated.*/\n#define APOP_SUBMATRIX(m, srow, scol, nrows, ncols, o) gsl_matrix apop_mm_##o = gsl_matrix_submatrix((m), (srow), (scol), (nrows),(ncols)).matrix;\\\ngsl_matrix * o = &( apop_mm_##o ); // Use \\ref Apop_subm. \n#define Apop_submatrix APOP_SUBMATRIX\n\n#define Apop_col_v(m, col, v) gsl_vector apop_vv_##v = ((col) == -1) ? (gsl_vector){} : gsl_matrix_column((m)->matrix, (col)).vector;\\\ngsl_vector * v = ((col)==-1) ? (m)->vector : &( apop_vv_##v ); // Use \\ref Apop_cv.\n\n#define Apop_row_v(m, row, v) Apop_matrix_row((m)->matrix, row, v) // Use \\ref Apop_rv.\n#define Apop_rows(d, rownum, len, outd) apop_data *outd = Apop_rs(d, rownum, len) // Use \\ref Apop_rs.\n#define Apop_row(d, row, outd) Apop_rows(d, row, 1, outd) // Use \\ref Apop_r.\n#define Apop_cols(d, colnum, len, outd) apop_data *outd = Apop_cs(d, colnum, len); // Use \\ref Apop_cs.\n/** \\endcond */ //End of Doxygen ignore.\n\n/** \\def Apop_row_tv(m, row_name, v)\n After this call, \\c v will hold a \\c gsl_vector view of an \\ref apop_data set \\c m. The view will consist only of the row with name \\c row_name.\n Unlike \\ref Apop_rv, the second argument is a row name, that I'll look up using \\ref apop_name_find, and the third is the name of the view to be generated.\n\\see Apop_rs, Apop_r, Apop_rv, Apop_row_t, Apop_mrv\n*/\n#define Apop_row_tv(m, row, v) gsl_vector apop_vv_##v = gsl_matrix_row((m)->matrix, apop_name_find((m)->names, row, 'r')).vector;\\\ngsl_vector * v = &( apop_vv_##v );\n\n/** \\def Apop_col_tv(m, col_name, v)\nAfter this call, \\c v will hold a \\c gsl_vector view of the \\ref apop_data set \\c m.\nThe view will consist only of the column with name \\c col_name.\nUnlike \\ref Apop_cv, the second argument is a column name, that I'll look up using \\ref apop_name_find, and the third is the name of the view to be generated.\n\\see Apop_cs, Apop_c, Apop_cv, Apop_col_t, Apop_mcv\n*/\n#define Apop_col_tv(m, col, v) gsl_vector apop_vv_##v = gsl_matrix_column((m)->matrix, apop_name_find((m)->names, col, 'c')).vector;\\\ngsl_vector * v = &( apop_vv_##v );\n\n/** \\def Apop_row_t(m, row_name, v)\n After this call, \\c v will hold an \\ref apop_data view of an \\ref apop_data set \\c m. The view will consist only of the row with name \\c row_name.\n Unlike \\ref Apop_r, the second argument is a row name, that I'll look up using \\ref apop_name_find, and the third is the name of the view to be generated.\n\\see Apop_rs, Apop_r, Apop_rv, Apop_row_tv, Apop_mrv\n*/\n#define Apop_row_t(d, rowname, outd) int apop_row_##outd = apop_name_find((d)->names, rowname, 'r'); Apop_rows(d, apop_row_##outd, 1, outd)\n\n/** \\def Apop_col_t(m, col_name, v)\n After this call, \\c v will hold a view of the \\ref apop_data set \\c m. The view will consist only of a \\c gsl_vector view of the column of the \\ref apop_data set \\c m with name \\c col_name.\n Unlike \\ref Apop_c, the second argument is a column name, that I'll look up using \\ref apop_name_find, and the third is the name of the view to be generated.\n\\see Apop_cs, Apop_c, Apop_cv, Apop_col_tv, Apop_mcv\n*/\n#define Apop_col_t(d, colname, outd) int apop_col_##outd = apop_name_find((d)->names, colname, 'c'); Apop_cols(d, apop_col_##outd, 1, outd)\n\n// The above versions relied on gsl_views, which stick to C as of 1989 CE.\n// Better to just create the views via designated initializers.\n\n\n/** \\def Apop_subm(data_to_view, srow, scol, nrows, ncols)\nGenerate a view of a submatrix within a \\c gsl_matrix. Like \\ref Apop_r, et al., the view is an automatically-allocated variable that is lost once the program flow leaves the scope in which it is declared.\n\n\\param data_to_view The root matrix\n\\param srow the first row (in the root matrix) of the top of the submatrix\n\\param scol the first column (in the root matrix) of the left edge of the submatrix\n\\param nrows number of rows in the submatrix\n\\param ncols number of columns in the submatrix\n\\return An automatically-allocated view of type \\c gsl_matrix.\n*/\n#define Apop_subm(matrix_to_view, srow, scol, nrows, ncols)( \\\n (!(matrix_to_view) \\\n || (matrix_to_view)->size1 < (srow)+(nrows) || (srow) < 0 \\\n || (matrix_to_view)->size2 < (scol)+(ncols) || (scol) < 0) ? NULL \\\n : &(gsl_matrix){.size1=(nrows), .size2=(ncols), \\\n .tda=(matrix_to_view)->tda, \\\n .data=gsl_matrix_ptr((matrix_to_view), (srow), (scol))} \\\n )\n\n/** Get a vector view of a single row of a \\ref gsl_matrix.\n\n\\param matrix_to_vew A \\ref gsl_matrix.\n\\param row An integer giving the row to be viewed.\n\\return A \\c gsl_vector view of the given row. The view is automatically allocated,\n and disappears as soon as the program leaves the scope in which it is declared.\n\nSee \\ref apop_vector_correlation for an example of use.\n\\see Apop_r, Apop_rv\n*/\n#define Apop_mrv(matrix_to_view, row) Apop_rv(&(apop_data){.matrix=matrix_to_view}, row)\n\n/** Get a vector view of a single column of a \\ref gsl_matrix.\n\n\\param matrix_to_vew A \\ref gsl_matrix.\n\\param row An integer giving the column to be viewed.\n\\return A \\c gsl_vector view of the given column. The view is automatically allocated,\n and disappears as soon as the program leaves the scope in which it is declared.\n\n\\code \ngsl_matrix *m = apop_query_to_data(\"select col1, col2, col3 from data\")->matrix;\nprintf(\"The correlation coefficient between columns two \"\n \"and three is %g.\\n\", apop_vector_correlation(Apop_mcv(m, 2), Apop_mcv(m, 3)));\n\\endcode \n\n\\see Apop_r, Apop_cv\n*/\n#define Apop_mcv(matrix_to_view, col) Apop_cv(&(apop_data){.matrix=matrix_to_view}, col)\n\n/** \\def Apop_rv(d, row)\nA macro to generate a temporary one-row view of the matrix in an \\ref apop_data set \\c d, pulling out only\nrow \\c row. The view is a \\c gsl_vector set.\n\n\\code\ngsl_vector *v = Apop_rv(your_data, i);\n\nfor (int i=0; i< your_data->matrix->size1; i++)\n printf(\"Σ_%i = %g\\n\", i, apop_vector_sum(Apop_r(your_data, i)));\n\\endcode\n\nThe view is automatically allocated, and disappears as soon as the program leaves the scope in which it is declared.\n\\see Apop_r, Apop_rv, Apop_row_tv, Apop_row_t, Apop_mrv\n*/\n#define Apop_rv(data_to_view, row) ( \\\n ((data_to_view) == NULL || (data_to_view)->matrix == NULL \\\n || (data_to_view)->matrix->size1 <= (row) || (row) < 0) ? NULL \\\n : &(gsl_vector){.size=(data_to_view)->matrix->size2, \\\n .stride=1, .data=gsl_matrix_ptr((data_to_view)->matrix, (row), 0)} \\\n )\n\n/** \\def Apop_cv(d, col)\nA macro to generate a temporary one-column view of the matrix in an \\ref apop_data\nset \\c d, pulling out only column \\c col. The view is a \\c gsl_vector set.\n\nAs usual, column -1 is the vector element of the \\ref apop_data set.\n\n\\code\ngsl_vector *v = Apop_cv(your_data, i);\n\nfor (int i=0; i< your_data->matrix->size2; i++)\n printf(\"Σ_%i = %g\\n\", i, apop_vector_sum(Apop_c(your_data, i)));\n\\endcode\n\nThe view is automatically allocated, and disappears as soon as the program leaves the\nscope in which it is declared.\n\n\\see Apop_cs, Apop_c, Apop_col_tv, Apop_col_t, Apop_mcv\n*/\n#define Apop_cv(data_to_view, col) ( \\\n !(data_to_view) ? NULL \\\n : (col)==-1 ? (data_to_view)->vector \\\n : (!(data_to_view)->matrix \\\n || (data_to_view)->matrix->size2 <= (col) || ((int)(col)) < -1) ? NULL \\\n : &(gsl_vector){.size=(data_to_view)->matrix->size1, \\\n .stride=(data_to_view)->matrix->tda, .data=gsl_matrix_ptr((data_to_view)->matrix, 0, (col))} \\\n )\n\n/** \\cond doxy_ignore */\n/* Not (yet) for public use. */\n#define Apop_subvector(v, start, len) ( \\\n ((v) == NULL || (v)->size < ((start)+(len)) || (start) < 0) ? NULL \\\n : &(gsl_vector){.size=(len), .stride=(v)->stride, .data=(v)->data+(start*(v)->stride)})\n/** \\endcond */\n\n/** \\def Apop_rs(d, row, len)\nA macro to generate a temporary view of \\ref apop_data set \\c d pulling only certain rows, beginning at row \\c row\nand having height \\c len. \n\nThe view is automatically allocated, and disappears as soon as the program leaves the scope in which it is declared.\n\\see Apop_r, Apop_rv, Apop_row_tv, Apop_row_t, Apop_mrv\n*/\n#define Apop_rs(d, rownum, len)( \\\n (!(d) || (rownum) < 0) ? NULL \\\n : &(apop_data){ \\\n .names= ( !((d)->names) ? NULL : \\\n &(apop_name){ \\\n .title = (d)->names->title, \\\n .vector = (d)->names->vector, \\\n .col = (d)->names->col, \\\n .row = ((d)->names->row && (d)->names->rowct > (rownum)) ? &((d)->names->row[rownum]) : NULL, \\\n .texthash = (d)->names->texthash, \\\n .rowhash = ((d)->names->rowhash && (d)->names->rowct > (rownum)) ? &((d)->names->rowhash[rownum]) : NULL, \\\n .colhash = (d)->names->colhash, \\\n .text = (d)->names->text, \\\n .colct = (d)->names->colct, \\\n .rowct = (d)->names->row ? (GSL_MIN(1, GSL_MAX((d)->names->rowct - (int)(rownum), 0))) \\\n : 0, \\\n .textct = (d)->names->textct }), \\\n .vector= Apop_subvector((d->vector), (rownum), (len)), \\\n .matrix = Apop_subm(((d)->matrix), (rownum), 0, (len), (d)->matrix?(d)->matrix->size2:0), \\\n .weights = Apop_subvector(((d)->weights), (rownum), (len)), \\\n .textsize[0]=(d)->textsize[0]> (rownum)+(len)-1 ? (len) : 0, \\\n .textsize[1]=(d)->textsize[1], \\\n .text = (d)->text ? &((d)->text[rownum]) : NULL, \\\n })\n\n\n/** \\def Apop_cs(d, col, len)\nA macro to generate a temporary view of \\ref apop_data set \\c d including only certain columns, beginning at column \\c col and having length \\c len. \n\nThe view is automatically allocated, and disappears as soon as the program leaves the scope in which it is declared.\n\\see Apop_c, Apop_cv, Apop_col_tv, Apop_col_t, Apop_mcv\n*/\n#define Apop_cs(d, colnum, len) ( \\\n (!(d)||!(d)->matrix || (d)->matrix->size2 <= (colnum)+(len)-1) \\\n ? NULL \\\n : &(apop_data){ \\\n .vector= NULL, \\\n .weights= (d)->weights, \\\n .matrix = Apop_subm((d)->matrix, 0, colnum, (d)->matrix->size1, (len)),\\\n .textsize[0] = 0, \\\n .textsize[1] = 0, \\\n .text = NULL, \\\n .names= (d)->names ? &(apop_name){ \\\n .title = (d)->names->title, \\\n .vector = NULL, \\\n .row = (d)->names->row, \\\n .col = ((d)->names->col && (d)->names->colct > colnum) ? &((d)->names->col[colnum]) : NULL, \\\n .text = NULL, \\\n .texthash = NULL, \\\n .rowhash = (d)->names->rowhash, \\\n .colhash = ((d)->names->colhash && (d)->names->colct > (colnum)) ? &((d)->names->colhash[colnum]) : NULL, \\\n .rowct = (d)->names->rowct, \\\n .colct = (d)->names->col ? (GSL_MIN(len, GSL_MAX((d)->names->colct - colnum, 0))) \\\n : 0, \\\n .textct = (d)->names->textct } : NULL \\\n })\n\n/** \\def Apop_r(d, row)\nA macro to generate a temporary one-row view of \\ref apop_data set \\c d, pulling out only\nrow \\c row. The view is also an \\ref apop_data set, with names and other decorations.\n\\code\n//pull a single row\napop_data *v = Apop_r(your_data, 7);\n\n//or loop through a sequence of one-row data sets.\napop_model *std = apop_model_set_parameters(apop_normal, 0, 1);\nfor (int i=0; i< your_data->matrix->size1; i++)\n printf(\"Std Normal CDF up to observation %i is %g\\n\",\n i, apop_cdf(Apop_r(your_data, i), std));\n\\endcode\n\nThe view is automatically allocated, and disappears as soon as the program leaves the\nscope in which it is declared.\n\\see Apop_rs, Apop_row_v, Apop_row_tv, Apop_row_t, Apop_mrv\n*/\n#define Apop_r(d, rownum) Apop_rs(d, rownum, 1)\n\n/** \\def Apop_c(d, col)\nA macro to generate a temporary one-column view of \\ref apop_data set \\c d, pulling out only\ncolumn \\c col. \nAfter this call, \\c outd will be a pointer to this temporary\nview, that you can use as you would any \\ref apop_data set.\n\\see Apop_cs, Apop_cv, Apop_col_tv, Apop_col_t, Apop_mcv\n*/\n#define Apop_c(d, col) Apop_cs(d, col, 1)\n\n/** \\cond doxy_ignore */\n#define APOP_COL Apop_col\n#define apop_col Apop_col\n#define APOP_COL_T Apop_col_t\n#define apop_col_t Apop_col_t\n#define APOP_COL_TV Apop_col_tv\n#define apop_col_tv Apop_col_tv\n\n#define APOP_ROW Apop_row\n#define apop_row Apop_row\n#define APOP_COLS Apop_cols\n#define apop_cols Apop_cols\n#define APOP_COL_V Apop_col_v\n#define apop_col_v Apop_col_v\n#define APOP_ROW_V Apop_row_v\n#define apop_row_v Apop_row_v\n#define APOP_ROWS Apop_rows\n#define apop_rows Apop_rows\n#define Apop_data_row Apop_row #deprecated\n#define APOP_ROW_T Apop_row_t\n#define apop_row_t Apop_row_t\n#define APOP_ROW_TV Apop_row_tv\n#define apop_row_tv Apop_row_tv\n\n/** Deprecated. Use Apop_mrv */\n#define Apop_matrix_row(m, row, v) gsl_vector apop_vv_##v = gsl_matrix_row((m), (row)).vector;\\\ngsl_vector * v = &( apop_vv_##v );\n\n/* Deprecated. Use Apop_mcv */\n#define Apop_matrix_col(m, col, v) gsl_vector apop_vv_##v = gsl_matrix_column((m), (col)).vector;\\\ngsl_vector * v = &( apop_vv_##v );\n\n#define APOP_MATRIX_ROW Apop_matrix_row \n#define apop_matrix_row Apop_matrix_row \n#define APOP_MATRIX_COL Apop_matrix_col \n#define apop_matrix_col Apop_matrix_col \n/** \\endcond */\n\n\nlong double apop_vector_sum(const gsl_vector *in);\ndouble apop_vector_var_m(const gsl_vector *in, const double mean);\nApop_var_declare( double apop_vector_correlation(const gsl_vector *ina, const gsl_vector *inb, const gsl_vector *weights) )\ndouble apop_vector_kurtosis(const gsl_vector *in);\ndouble apop_vector_skew(const gsl_vector *in);\n\n#define apop_sum apop_vector_sum\n#define apop_var apop_vector_var\n#define apop_mean apop_vector_mean\n\n //////database utilities\n\nApop_var_declare( int apop_table_exists(char const *name, char remove) )\n\nint apop_db_open(char const *filename);\nApop_var_declare( int apop_db_close(char vacuum) )\n\nint apop_query(const char *q, ...) __attribute__ ((format (printf,1,2)));\napop_data * apop_query_to_text(const char * fmt, ...) __attribute__ ((format (printf,1,2)));\napop_data * apop_query_to_data(const char * fmt, ...) __attribute__ ((format (printf,1,2)));\napop_data * apop_query_to_mixed_data(const char *typelist, const char * fmt, ...) __attribute__ ((format (printf,2,3)));\ngsl_vector * apop_query_to_vector(const char * fmt, ...) __attribute__ ((format (printf,1,2)));\ndouble apop_query_to_float(const char * fmt, ...) __attribute__ ((format (printf,1,2)));\n\nint apop_data_to_db(const apop_data *set, const char *tabname, char);\n\n\n //////Settings groups\n\n //Part I: macros and fns for getting/setting settings groups and elements\n\n/** \\cond doxy_ignore */\nvoid * apop_settings_get_grp(apop_model *m, char *type, char fail);\nvoid apop_settings_remove_group(apop_model *m, char *delme);\nvoid apop_settings_copy_group(apop_model *outm, apop_model *inm, char *copyme);\nvoid *apop_settings_group_alloc(apop_model *model, char *type, void *free_fn, void *copy_fn, void *the_group);\napop_model *apop_settings_group_alloc_wm(apop_model *model, char *type, void *free_fn, void *copy_fn, void *the_group);\n/** \\endcond */ //End of Doxygen ignore.\n\n/** Retrieves a settings group from a model. See \\ref Apop_settings_get\nto just pull a single item from within the settings group.\n\nThis macro returns NULL if a group of type \\c type_settings isn't found attached\nto model \\c m, so you can easily put it in a conditional like\n \\code \n if (!apop_settings_get_group(m, \"apop_ols\")) ...\n \\endcode\n\n\\param m An \\ref apop_model\n\\param type A string giving the type of the settings group you are retrieving. E.g., for an \\ref apop_mle_settings group, use only \\c apop_mle.\n\\return A void pointer to the desired struct (or \\c NULL if not found).\n*/\n#define Apop_settings_get_group(m, type) apop_settings_get_grp(m, #type, 'c')\n\n/** Removes a settings group from a model's list. \n \n\\li If the so-named group is not found, do nothing.\n*/\n#define Apop_settings_rm_group(m, type) apop_settings_remove_group(m, #type)\n\n/** Add a settings group. The first two arguments (the model you are\nattaching to and the settings group name) are mandatory, and then you\ncan use the \\ref designated syntax to specify default values (if any).\n\\return A pointer to the newly-prepped group.\n\nSee \\ref modelsettings, \\ref maxipage, or \\ref Apop_settting_set for examples.\n\n\\li If a settings group of the given type is already attached to the model, \nthe previous version is removed. Use \\ref Apop_settings_get to check whether a group\nof the given type is already attached to a model, and \\ref Apop_settings_set to modify\nan existing group.\n*/\n#define Apop_settings_add_group(model, type, ...) \\\n apop_settings_group_alloc(model, #type, type ## _settings_free, type ## _settings_copy, type ##_settings_init ((type ## _settings) {__VA_ARGS__}))\n\n/** Copy a model and add a settings group. Useful for models that require a settings group to function. See \\ref Apop_settings_add_group.\n\n\\return A pointer to the newly-prepped model.\n*/\n#define apop_model_copy_set(model, type, ...) \\\n apop_settings_group_alloc_wm(apop_model_copy(model), #type, type ## _settings_free, type ## _settings_copy, type ##_settings_init ((type ## _settings) {__VA_ARGS__}))\n\n\n/** This is the complement to \\ref apop_model_set_parameters, for those models that are\n set up by adding settings group, rather than filling in a list of parameters.\n\nFor example, the \\ref apop_kernel_density model is built by adding a \\ref apop_kernel_density_settings group. From the example on the \\ref apop_kernel_density page:\n\n\\code\napop_model *k2 = apop_model_set_settings(apop_kernel_density,\n .base_data=d,\n .set_fn = set_uniform_edges,\n .kernel = apop_uniform);\n\\endcode\n\nThe name of the model and the settings group to be built must match, which is the case\nfor many model transformations, including \\ref apop_dconstrain and \\ref apop_cross. If the names do not match, use \\ref apop_model_copy_set.\n*/\n#define Apop_model_set_settings(model, ...) \\\n apop_settings_group_alloc_wm(apop_model_copy(model), #model, model ## _settings_free, model ## _settings_copy, model ##_settings_init ((model ## _settings) {__VA_ARGS__}))\n\n#define apop_model_set_settings Apop_model_set_settings\n\n/** Retrieves a setting from a model. See \\ref Apop_settings_get_group to pull the entire group.\n\n\\param model An \\ref apop_model.\n\\param type A string giving the type of the settings group you are retrieving, without the \\c _settings ending. E.g., for an \\ref apop_mle_settings group, use \\c apop_mle.\n\\param setting The struct element you want to retrieve.\n*/\n#define Apop_settings_get(model, type, setting) \\\n (((type ## _settings *) apop_settings_get_grp(model, #type, 'f'))->setting)\n\n/** Modifies a single element of a settings group to the given value. \n\nFor example,\n\\code\n//set up a mixture of two Normals. This function initializes an apop_mixture_settings group\napop_model *mix = apop_model_mixture(apop_model_copy(apop_normal), apop_model_copy(apop_normal));\n\n//Add an apop_mle_settings group to specify the search strategy\nApop_settings_add_group(mix, apop_mle, .starting_pt=(double[]){.5, .5, 50, 5, 80, 5},\n .step_size=3, .tolerance=1e-6);\n\n//The mix model now has apop_mle and apop_mixture settings groups attached. Modify them:\nApop_settings_set(mix, apop_mixture, find_weights, 'y'); //Search for optimal mixture weights\nApop_settings_set(mix, apop_mle, method, \"NM simplex\"); //Nelder-Mead simplex algorithm\napop_model *optimal_mix = apop_estimate(input_data, mix); //Everything is set up, so do the search.\n\\endcode\n\n\\li If model==NULL, fails silently. \n\\li If model!=NULL but the given settings group is not found attached to the model, set model->error='s'.\n*/\n#define Apop_settings_set(model, type, setting, data) \\\n do { \\\n if (!(model)) continue; /* silent fail. */ \\\n type ## _settings *apop_tmp_settings = apop_settings_get_grp(model, #type, 'c'); \\\n Apop_stopif(!apop_tmp_settings, (model)->error='s', 0, \"You're trying to modify a setting in \" \\\n #model \"'s setting group of type \" #type \" but that model doesn't have such a group.\"); \\\n apop_tmp_settings->setting = (data); \\\n } while (0);\n\n/** \\cond doxy_ignore */\n#define Apop_settings_add Apop_settings_set\n#define APOP_SETTINGS_ADD Apop_settings_set\n#define apop_settings_set Apop_settings_set\n#define APOP_SETTINGS_GET Apop_settings_get\n#define apop_settings_get Apop_settings_get\n#define APOP_SETTINGS_ADD_GROUP Apop_settings_add_group\n#define apop_settings_add_group Apop_settings_add_group\n#define APOP_SETTINGS_GET_GROUP Apop_settings_get_group\n#define apop_settings_get_group Apop_settings_get_group\n#define APOP_SETTINGS_RM_GROUP Apop_settings_rm_group\n#define apop_settings_rm_group Apop_settings_rm_group\n#define Apop_model_copy_set apop_model_copy_set\n\n//deprecated:\n#define Apop_model_add_group Apop_settings_add_group\n\n/** \\endcond */ //End of Doxygen ignore.\n\n/** Put this in your header file to declare the init, copy, and\nfree functions for ysg_settings. Of course, these functions will also have to be defined\nin a .c file using \\ref Apop_settings_init, \\ref Apop_settings_copy, and \\ref Apop_settings_free. */\n#define Apop_settings_declarations(ysg) \\\n ysg##_settings * ysg##_settings_init(ysg##_settings); \\\n void * ysg##_settings_copy(ysg##_settings *); \\\n void ysg##_settings_free(ysg##_settings *);\n\n/** A convenience macro for declaring the initialization function for a new settings group.\nSee \\ref settingswriting for details and an example.\n*/\n#define Apop_settings_init(name, ...) \\\n name##_settings *name##_settings_init(name##_settings in) { \\\n name##_settings *out = malloc(sizeof(name##_settings)); \\\n *out = in; \\\n __VA_ARGS__; \\\n return out; \\\n }\n\n/** \\cond doxy_ignore */\n#define Apop_varad_set(var, value) (out)->var = (in).var ? (in).var : (value);\n/** \\endcond */\n\n/** A convenience macro for declaring the copy function for a new settings group.\nSee \\ref settingswriting for details and an example.\n*/\n#define Apop_settings_copy(name, ...) \\\n void * name##_settings_copy(name##_settings *in) {\\\n name##_settings *out = malloc(sizeof(name##_settings)); \\\n *out = *in; \\\n __VA_ARGS__; \\\n return out; \\\n }\n\n/** A convenience macro for declaring the delete function for a new settings group.\nSee \\ref settingswriting for details and an example.\n*/\n#define Apop_settings_free(name, ...) \\\n void name##_settings_free(name##_settings *in) {\\\n __VA_ARGS__; \\\n free(in); \\\n }\n\n //Part II: the details of extant settings groups.\n\n\n/** The settings for maximum likelihood estimation (including simulated annealing). */\ntypedef struct{\n double *starting_pt; /**< An array of doubles (e.g., (double*){2,4,6,8}) suggesting a starting point. \n If NULL, use an all-ones vector. If \\c startv is a \\c gsl_vector\n and is not a view of a matrix, use .starting_pt=startv->data.*/\n char *method; /**< The method to be used for the optimization. All strings are case-insensitive.\n\n \n\n Name Notes\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n If Newton proposes stepping outside of a certain interval, use an alternate method. See the GSL manual for discussion.\n\n\n
String
\"NM simplex\" Nelder-Mead simplex Does not use gradients at all. Can sometimes get stuck.
\"FR cg\" Conjugate gradient (Fletcher-Reeves) (default) CG methods use derivatives. The converge to the optimum of a quadratic function in one step; performance degrades as the objective digresses from quadratic.
\"BFGS cg\" Broyden-Fletcher-Goldfarb-Shanno conjugate gradient
\"PR cg\" Polak-Ribiere conjugate gradient
\"Annealing\" \\ref simanneal \"simulated annealing\" Slow but works for objectives of arbitrary complexity, including stochastic objectives.
\"Newton\" Newton's method Search by finding a root of the derivative. Expects that gradient is reasonably well-behaved.
\"Newton hybrid\" Newton's method/gradient descent hybrid Find a root of the derivative via the Hybrid method
\"Newton hybrid no scale\" Newton's method/gradient descent hybrid with spherical scale As above, but use a simplified trust region.
*/\n double step_size, /**< The initial step size. */\n tolerance, /**< The precision the minimizer uses in its stopping rule. Only vaguely related to the precision of the actual MLE.*/\ndelta;\n int max_iterations; /**< Ignored by simulated annealing. Other methods halt (and set the \\c \"status\" element of the output estimate's info page) if\n they do this many iterations without finding an optimum. */\n int verbose; /**<\tGive status updates as we go. This is orthogonal to the \n apop_opts.verbose setting. */\n double dim_cycle_tolerance; /**< If zero (the default), the usual procedure.\n If \\f$>0\\f$, cycle across dimensions: fix all but the first dimension at the starting\n point, optimize only the first dim. Then fix the all but the second dim, and optimize the\n second dim. Continue through all dims, until the log likelihood at the outset of one cycle\n through the dimensions is within this amount of the previous cycle's log likelihood. There\n will be at least two cycles.\n */\n//simulated annealing (also uses step_size);\n int n_tries, iters_fixed_T;\n double k, t_initial, mu_t, t_min ;\n gsl_rng *rng;\n apop_data **path; /**< If not \\c NULL, record each vector tried by the optimizer as one row of this \\ref apop_data set.\n Each row of the \\c matrix element holds the vector tried; the corresponding element in the \\c vector is the evaluated value at that vector (after out-of-constraints penalties have been subtracted).\n A new \\ref apop_data set is allocated at the pointer you send in. This data set has no names; add them as desired. For a sample use, see \\ref maxipage.\n*/\n} apop_mle_settings;\n\n/** Settings for least-squares type models such as \\ref apop_ols or \\ref apop_iv */\ntypedef struct {\n int destroy_data; /**< If \\c 'y', then the input data set may be normalized or otherwise mangled. */\n apop_data *instruments; /**< Use for the \\ref apop_iv regression, qv. */\n char want_cov; /**< Deprecated. Please use \\ref apop_parts_wanted_settings. */\n char want_expected_value; /**< Deprecated. Please use \\ref apop_parts_wanted_settings. */\n apop_model *input_distribution; /**< The distribution of \\f$P(Y|X)\\f$ is specified by the model holding this struct, but the distribution of \\f$X\\f$ needs to be specified as well for any calculation of \\f$P(Y)\\f$. See the notes in the RNG section of the \\ref apop_ols documentation. */\n} apop_lm_settings;\n\n/** The default is for the estimation routine to give some auxiliary information,\n such as a covariance matrix, predicted values, and common hypothesis tests.\n Some uses of a model depend on these items, but if they are a waste\n of time for your purposes, this settings group gives a quick way to bypass them all.\n\n Adding this settings group to your model without changing any default values---\n \\code\n Apop_model_add_group(your_model, apop_parts_wanted);\n \\endcode\n ---will turn off all of the auxiliary calculations covered, because the default value\n for all the switches is 'n', indicating that all elements are not wanted.\n\n From there, you can change some of the default 'n's to 'y's to retain some but not all auxiliary elements. If you just want the parameters themselves and the covariance matrix:\n \\code\n Apop_model_add_group(your_model, apop_parts_wanted, .covariance='y');\n \\endcode\n\n \\li Not all models support this, although the models with especially compute-intensive\n auxiliary info do (e.g., the maximum likelihood estimation system). Check the model's documentation. \n\n \\li Tests may depend on covariance, so .covariance='n', .tests='y' may be \n treated as .covariance='y', .tests='y'.\n*/\ntypedef struct {\n //init/copy/free are in apop_mle.c\n char covariance; /*< If 'y', calculate the covariance matrix. Default 'n'. */\n char predicted;/*< If 'y', calculate the predicted values. This is typically as many\n items as rows in your data set. Default 'n'. */\n char tests;/*< If 'y', run any hypothesis tests offered by the model's estimation routine. Default 'n'. */\n char info;/*< If 'y', add an info table with elements such as log likelihood or AIC. Default 'n'. */\n} apop_parts_wanted_settings;\n\n/** For use by \\ref apop_cdf when the CDF is generated via Monte Carlo methods. */\ntypedef struct {\n int draws; /**< For random draw methods, how many draws? Default: 10,000.*/\n gsl_rng *rng; /**< For random draw methods. See \\ref apop_rng_get_thread on the default. */\n gsl_matrix *draws_made; /**< A store of random draws used to calcuate the CDF. Need only be generated once, and so stored here. */\n int *draws_refcount; /**< For internal use.*/\n} apop_cdf_settings;\n\n\n/** Settings for getting parameter models (i.e. the distribution of parameter estimates) */\ntypedef struct {\n apop_model *base;\n int index;\n gsl_rng *rng;\n int draws;\n} apop_pm_settings;\n\n\n/** Settings to accompany the \\ref apop_pmf. */\ntypedef struct {\n gsl_vector *cmf; /**< A cumulative mass function, for the purposes of making random draws.*/\n char draw_index; /**< If \\c 'y', then draws from the PMF return the integer index of the row drawn. \n If \\c 'n' (the default), then return the data in the vector/matrix elements of the data set. */\n long double total_weight; /**< Keep the total weight, in case the input weights aren't normalized to sum to one. */\n int *cmf_refct; /**< For internal use, so I can garbage-collect the CMF when needed. */\n} apop_pmf_settings;\n\n\n/** Settings for the \\ref apop_kernel_density model. */\ntypedef struct{\n apop_data *base_data; /**< The data that will be smoothed by the KDE. */\n apop_model *base_pmf; /**< I actually need the data in a \\ref apop_pmf. You can give\n that to me explicitly, or I can wrap the .base_data in a PMF. */\n apop_model *kernel; /**< The distribution to be centered over each data point. Default, \n \\ref apop_normal with std dev 1. */\n void (*set_fn)(apop_data*, apop_model*); /**< The function I will use for each data\n point to center the kernel over each point.\n Default: set the upper-left element of the parameter set to the upper-left scalar in the data:\n apop_data_set(m->parameters, .val= apop_data_get(in));.\n */\n int own_pmf, own_kernel; /**< For internal use only. */\n}apop_kernel_density_settings;\n\nstruct apop_mcmc_settings;\n\n/** A proposal distribution for \\ref apop_mcmc_settings and its accompanying functions and\ninformation. By default, these will be \\ref apop_multivariate_normal models. The \\c\nstep_fn and \\c adapt_fn have to be written around the model and your preferences.\nFor the defaults, the step function recenters the mean of the distribution around the\nlast accepted proposal, and the adapt function widens \\f$\\Sigma\\f$ for the Normal if the\naccept rate is too low; narrows it if the accept rate is too large.\n\nYou may provide an array of proposals. The length of the list of proposals\nmust match the number of chunks, as per the \\c gibbs_chunks setting in the \\ref\napop_mcmc_settings group that the array of proposals is a part of. Each proposal must\nbe initialized to include all elements, and the step and adapt functions probably have\nto be written anew for each type of model.\n*/\ntypedef struct apop_mcmc_proposal_s {\n apop_model *proposal; /**< The distribution from which test parameters will be\n drawn. After getting the draw using the \\c draw method of the proposal, the base\n model's \\c parameters element is filled using \\ref apop_data_fill.\n If \\c NULL, \\ref apop_model_metropolis will use a Multivariate Normal with the\n appropriate dimension, mean zero, and covariance matrix I. If not \\c NULL, be sure to\n parameterize your model with an initial position. */\n\n void (*step_fn)(double const *, struct apop_mcmc_proposal_s*, struct apop_mcmc_settings *); /**< Modifies the parameters of the\n proposal distribution given a successful draw. Typically, this function writes the\n drawn data point to the parameter set. If the draw is a scalar, the default\n function sets the 0th element of the model's \\c parameter set with the draw\n (works for the \\ref apop_normal and other models). If the draw has multiple\n dimensions, they are all copied to the parameter set, which must have the same\n size. */\n\n int (*adapt_fn)(struct apop_mcmc_proposal_s *ps, struct apop_mcmc_settings *ms); /**< Called\n every step, to adapt the proposal distribution using information to this point in\n the chain. */\n\n int accept_count, reject_count; /**< If there are multiple \\ref apop_mcmc_proposal_s structs for \n multiple chunks, These count accepts/rejects for\n this chunk. The \\ref apop_mcmc_settings group has\n a total for the aggregate across all chunks. */\n} apop_mcmc_proposal_s;\n\n/** Method settings for a model to be put through Bayesian updating. */\ntypedef struct apop_mcmc_settings {\n apop_data *data;\n long int periods; /**< For how many steps should the MCMC chain run? */\n double burnin; /**< What percentage of the periods should be ignored\n as initialization. That is, this is a number between zero and one. */\n int histosegments; /**< If outputting a binned PMF, how many segments should it have? */\n double last_ll; /**< If you have already run MCMC, the last log likelihood in the chain.*/\n apop_model *pmf; /**< If you have already run MCMC, I keep a pointer to the model\n so far here. Use \\ref apop_model_metropolis_draw to get one more draw.*/\n apop_model *base_model; /**< The model you provided with a \\c log_likelihood or\n \\c p element (which need not sum to one). You do not have to set this: if it is\n \\c NULL on input to \\ref apop_model_metropolis, I will fill it in.*/\n apop_mcmc_proposal_s *proposals; /**< The list of proposals. You can probably use\n the default of adaptive multivariate normals. See the \\ref apop_mcmc_proposal_s\n struct for details. */\n int proposal_count; /**< The number of proposal sets; see \\c gibbs_chunks below. */\n double target_accept_rate; /**< The desired acceptance rate, for use by adaptive proposals. Default: .35 */\n int accept_count; /**< After calling \\ref apop_model_metropolis, this will have the number of accepted proposals.*/\n int reject_count; /**< After calling \\ref apop_model_metropolis, this will have the number of rejected proposals.*/\n char gibbs_chunks; /**< See the \\ref apop_model_metropolis documentation for discussion.\n \n \\c 'a': One step draws and accepts/rejects all parameters as a unit
\n\n \\c 'b': draw in blocks: the vector is a block, the matrix\n is a separate block, the weights are a separate\n block, and so on through every page of the model\n parameters. Each block of parameters is drawn and\n accepted/rejected as a unit.
\n\n \\c '1': draw each parameter and accept/reject separately. One\n MCMC step consists of a set of draws for every\n parameter.
*/\n size_t *block_starts; /**< For internal use */\n int block_count, proposal_is_cp; /**< For internal use. */\n\n char start_at; /**< If \\c '1' (the default), start with a first proposal of all\n 1s. Even when this is a far-from-useful starting point, MCMC typically does a good\n job of crawling to better spots early in the chain.
\n The default when this is unset is to start at the \\c parameters of the \\ref apop_model sent in to \\ref\n apop_model_metropolis.*/\n void (*base_step_fn)(double const *, struct apop_mcmc_proposal_s*, struct apop_mcmc_settings *); /**< If an \\ref apop_mcmc_proposal_s struct has \\c NULL \\c step_fn, use this. If you don't want a step function, set this to a do-nothing function. */\n int (*base_adapt_fn)(struct apop_mcmc_proposal_s *ps, struct apop_mcmc_settings *ms); /**< If a \\ref apop_mcmc_proposal_s has \\c NULL \\c adapt_fn, use this. If you don't want an adapt function, set this to a do-nothing function.*/\n\n} apop_mcmc_settings;\n\n/** \\cond doxy_ignore */\n//Loess, including the old FORTRAN-to-C.\nstruct loess_struct {\n\tstruct {\n\t\tlong n, p;\n double *y, *x;\n\t\tdouble\t*weights;\n\t} in;\n\tstruct {\n\t double span;\n\t long degree;\n\t long normalize;\n\t long parametric[8];\n\t long drop_square[8];\n\t char *family;\n\t} model;\n\tstruct {\n\t char *surface;\n\t char *statistics;\n\t double cell;\n\t char *trace_hat;\n\t long iterations;\n\t} control;\n\tstruct {\n\t\tlong\t*parameter, *a;\n\t\tdouble\t*xi, *vert, *vval;\n\t} kd_tree;\n\tstruct {\n\t\tdouble\t*fitted_values;\n double *fitted_residuals;\n\t\tdouble enp, s;\n\t\tdouble one_delta, two_delta;\n\t\tdouble\t*pseudovalues;\n\t\tdouble\ttrace_hat;\n\t\tdouble\t*diagonal;\n\t\tdouble\t*robust;\n\t\tdouble *divisor;\n\t} out;\n};\n/** \\endcond */ //End of Doxygen ignore.\n\n/** The code for the loess system is based on FORTRAN code from 1988,\noverhauled in 1992, linked in to Apophenia in 2009. The structure that\ndoes all the work, then, is a \\c loess_struct that you should\nbasically take as opaque. \n\nThe useful settings from that struct re-appear in the \\ref\napop_loess_settings struct so you can set them directly, and then the\nsettings init function will copy your preferences into the working struct.\n\nThe documentation for the elements is cut/pasted/modified from Cleveland,\nGrosse, and Shyu.\n*/\ntypedef struct {\n apop_data *data;\n struct loess_struct lo_s; /**< \n\n.data: Mandatory. Your input data set.\n\n.lo_s.model.span:\tsmoothing parameter. Default is 0.75.\n\n.lo_s.model.degree: overall degree of locally-fitted polynomial. 1 is\n\t\tlocally-linear fitting and 2 is locally-quadratic fitting. Default is 2.\n\n.lo_s.normalize:\tShould numeric predictors\n\t\tbe normalized?\tIf \\c 'y' - the default - the standard normalization\n\t\tis used. If \\c 'n', no normalization is carried out.\n\n\\c .lo_s.model.parametric:\tfor two or more numeric predictors, this argument\n\t\tspecifies those variables that should be\n\t\tconditionally-parametric. The argument should be a logical\n\t\tvector of length \\c p, specified in the order of the predictor\n\t\tgroup ordered in \\c x. Default is a vector of 0's of length \\c p.\n\n\\c .lo_s.model.drop_square:\tfor cases with degree = 2, and with two or more\n\t\tnumeric predictors, this argument specifies those numeric\n\t\tpredictors whose squares should be dropped from the set of\n\t\tfitting variables. The method of specification is the same as\n\t\tfor parametric. Default is a vector of 0's of length p.\n\n\\c .lo_s.model.family: the assumed distribution of the errors. The values may be \n \"gaussian\" or \"symmetric\". The first value is the default.\n If the second value is specified, a robust fitting procedure is used.\n\n\\c lo_s.control.surface:\tdetermines whether the fitted surface is computed\n \"directly\" at all points or whether an \"interpolation\"\n method is used. The default, interpolation, is what most users should use\n\t\tunless special circumstances warrant.\n\n\\c lo_s.control.statistics:\tdetermines whether the statistical quantities are \n computed \"exactly\" or approximately, where \"approximate\"\n is the default. The former should only be used for testing the approximation in\n statistical development and is not meant for routine usage because computation\n time can be horrendous.\n\n \\c lo_s.control.cell: if interpolation is used to compute the surface,\n this argument specifies the maximum cell size of the k-d tree. Suppose k =\n floor(n*cell*span) where n is the number of observations. Then a cell is\n further divided if the number of observations within it is greater than or\n equal to k. default=0.2\n\n\\c lo_s.control.trace_hat: Options are \"approximate\", \"exact\", and \"wait.to.decide\".\t\n When lo_s.control.surface is \"approximate\", determines\n the computational method used to compute the trace of the hat\n matrix, which is used in the computation of the statistical\n quantities. If \"exact\", an exact computation is done; normally\n this goes quite fast on the fastest machines until n, the number\n of observations is 1000 or more, but for very slow machines,\n things can slow down at n = 300. If \"wait.to.decide\" is selected,\n then a default is chosen in loess(); the default is \"exact\" for\n n < 500 and \"approximate\" otherwise. If surface is \"exact\", an\n exact computation is always done for the trace. Set trace_hat to\n \"approximate\" for large dataset will substantially reduce the\n computation time.\n\n\\c lo_s.model.iterations:\tif family is \"symmetric\", the number of iterations \n of the robust fitting method. Default is 0 for\n lo_s.model.family = gaussian; 4 for family=symmetric.\n\n That's all you can set. Here are some output parameters:\n\n\\c fitted_values:\tfitted values of the local regression model\n\n\\c fitted_residuals:\tresiduals of the local regression fit\n\n \\c enp:\t\tequivalent number of parameters.\n\n \\c s:\t\testimate of the scale of the residuals.\n\n \\c one_delta:\ta statistical parameter used in the computation of standard errors.\n\n \\c two_delta:\ta statistical parameter used in the computation of standard errors.\n\n \\c pseudovalues:\tadjusted values of the response when robust estimation is used.\n\n\\c trace_hat:\ttrace of the operator hat matrix.\n\n \\c diagonal:\tdiagonal of the operator hat matrix.\n\n \\c robust:\t\trobustness weights for robust fitting.\n\n \\c divisor:\tnormalization divisor for numeric predictors.\n*/\n\n int want_predict_ci; /**< If \\c 'y' (the default), calculate the\n confidence bands for predicted values */\n double ci_level; /**< If running a prediction, the level at which\n to calculate the confidence interval. default: 0.95 */\n} apop_loess_settings;\n\n\n /** \\cond doxy_ignore */\ntypedef struct point { /* a point in the x,y plane */\n double x,y; /* x and y coordinates */\n double ey; /* exp(y-ymax+YCEIL) */\n double cum; /* integral up to x of rejection envelope */\n int f; /* is y an evaluated point of log-density */\n struct point *pl,*pr; /* envelope points to left and right of x */\n} POINT;\n\n/* This includes the envelope info and the metropolis steps. */\ntypedef struct { /* attributes of the entire rejection envelope */\n int cpoint; /* number of POINTs in current envelope */\n int npoint; /* max number of POINTs allowed in envelope */\n double ymax; /* the maximum y-value in the current envelope */\n POINT *p; /* start of storage of envelope POINTs */\n double *convex; /* adjustment for convexity */\n double metro_xprev; /* previous Markov chain iterate */\n double metro_yprev; /* current log density at xprev */\n} arms_state;\n /** \\endcond */\n\n/** For use with \\ref apop_arms_draw, to perform derivative-free adaptive rejection sampling with metropolis step. \n\nThat function generates default values for this struct if you do not attach one to the\nmodel beforehand, via a form like apop_model_add_group(your_model, apop_arms,\n.model=your_model, .xl=8, .xr =14);. If you initialize it manually via \\ref\napop_settings_add_group, the \\c model element is mandatory; you'll get a run-time\ncomplaint if you forget it.\n*/\ntypedef struct {\n double *xinit; /**< A double* giving starting values for x in ascending\n order, e.g., (double *){1, 10, 100}. . Default: -1,\n 0, 1. If this isn't \\c NULL, I need at least three items, and\n the length in \\c ninit. */\n double xl; /**< Left bound. If you don't give me one, I'll use min[min(xinit)/10, min(xinit)*10].*/\n double xr; /**< Right bound. If you don't give me one, I'll use max[max(xinit)/10, max(xinit)*10]. */\n double convex; /**< Adjustment for convexity */\n int ninit; /**< The length of \\c xinit.*/\n int npoint; /**< Maximum number of envelope points. I \\c malloc space for this many doubles at the outset. Default = 1e5. */\n char do_metro; /**< Set to \\c 'y' if the metropolis step is required (i.e.,\n if you're not sure if the function is log-concave).*/\n double xprev; /**< For internal use; please ignore. Previous value from Markov chain. */\n int neval; /**< On exit, the number of function evaluations performed */\n arms_state *state;\n apop_model *model; /**< The model from which to draw. Mandatory. Must have either a \\c log_likelihood or \\c p method.*/\n} apop_arms_settings;\n\n\n/** The settings to accompany the \\ref apop_cross model, representing the cross product of two models (or, via recursion, a list of models of arbitrary length).*/\ntypedef struct {\n char *splitpage; /**< The name of the page at which to split the data. If \\c NULL, I send the entire data set to both models as needed. */\n apop_model *model1; /**< The first model in the stack.*/\n apop_model *model2; /**< The second model.*/\n} apop_cross_settings;\n\ntypedef struct {\n apop_data *(*base_to_transformed)(apop_data*); /**< The function to transform the model from pre-transform space to post-transform space. */\n apop_data *(*transformed_to_base)(apop_data*); /**< The function to transform from post-transform space back to pre-transform space. If this function does not exist, using a Jacobian-based transformation is probably not mathematically correct. */\n double (*jacobian_to_base)(apop_data*); /**< The derivative of the \\c transformed_to_base function. */\n apop_model *base_model; /**< The pre-transformation model. */\n} apop_coordinate_transform_settings;/**< Settings for an \\ref apop_coordinate_transform model; see its documentation for notes and an example.\n*/\n\n/** For use with the \\ref apop_dconstrain model. See its documentation for an example. \n*/\ntypedef struct {\n apop_model *base_model; /**< The model, before constraint. */\n double (*constraint)(apop_data *, apop_model *); /**< The constraint. Return 1 if the data is in the constraint; zero if out. */\n double (*scaling)(apop_model *); /**< Optional. Return the percent of the model density inside the constraint. */\n gsl_rng *rng; /**< If you don't provide a \\c scaling function, I calculate the in-constraint model density via random draws.\n If no \\c rng is provided, I use a default RNG; see \\ref apop_rng_get_thread. */\n double scale; /**< After the scaling has been calculated, store it here. If you change the parameters of your base model,\n set this to zero to have the scaling recalculated. */\n gsl_vector *last_params; /**< The parameters used to calculate \\c scale. If these change, recalculate. */\n int draw_ct; /**< How many draws to make for calculating the in-constraint model density via random draws. Current default: 1e4. */\n int refct; /**< For internal use. */\n} apop_dconstrain_settings;\n\ntypedef struct {\n apop_model *generator_m;\n apop_model *ll_m;\n int draw_ct;\n} apop_composition_settings;/**< All of the elements of this struct should be considered private.*/\n\n/** For mixture distributions, typically set up using \\ref apop_model_mixture. See\n\\ref apop_mixture for discussion. Please consider all elements but \\c model_list and \\c\nweights as private and subject to change. See the examples for use of these elements. \n*/\ntypedef struct {\n gsl_vector *weights; /**< The likelihood of a draw from each component. Default is equal likelihood\n for each mixture element. Or set this to a weight vector of your choosing, or set\n find_weights='y' and have apop_estimate find optimal weights. */\n apop_model **model_list; /**< A \\c NULL-terminated list of component models. */\n int model_count;\n int *param_sizes; /**< The number of parameters for each model. Useful for unpacking the params. */\n apop_model *cmf; /**< For internal use by the draw method. */\n int *cmf_refct; /**< For internal use, so I can garbage-collect the CMF when needed. */\n char find_weights; /**< By default, weights are fixed. Set this b \\c 'y' to allow \\ref apop_estimate to\n use an EM algorithm to find the optimal weights.\n See the documentation for \\ref apop_mixture for details. */\n gsl_vector *next_weights; /**< For internal use.*/\n} apop_mixture_settings;\n\n //Models built via call to apop_model_copy_set.\n\n#define apop_model_dcompose(...) Apop_model_set_settings(apop_composition, __VA_ARGS__)\n#define apop_model_dconstrain(...) Apop_model_set_settings(apop_dconstrain, __VA_ARGS__)\n#define apop_model_coordinate_transform(...) Apop_model_set_settings(apop_coordinate_transform, __VA_ARGS__)\n\n//Doxygen drops whatever is after these declarations, so I put them last.\nApop_settings_declarations(apop_lm)\nApop_settings_declarations(apop_pm)\nApop_settings_declarations(apop_pmf)\nApop_settings_declarations(apop_mle)\nApop_settings_declarations(apop_cdf)\nApop_settings_declarations(apop_arms)\nApop_settings_declarations(apop_mcmc)\nApop_settings_declarations(apop_loess)\nApop_settings_declarations(apop_cross)\nApop_settings_declarations(apop_mixture)\nApop_settings_declarations(apop_dconstrain)\nApop_settings_declarations(apop_composition)\nApop_settings_declarations(apop_parts_wanted)\nApop_settings_declarations(apop_kernel_density)\nApop_settings_declarations(apop_coordinate_transform)\n\n#ifdef\t__cplusplus\n}\n#endif\n\n/** @} */ //End doxygen's all_public grouping\n\n//Part of the intent of a convenience header like this is that you\n//don't have to remember what else you're including. So here are \n//some other common GSL headers:\n#include \n#include \n#include \n#include \n#include \n#include \n" }, { "path": "apop_arms.c", "content": "/** \\file \n adaptive rejection metropolis sampling */\n\n/** (C) Wally Gilks; see documentation below for details.\n Adaptations for Apophenia (c) 2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n\n#define XEPS 0.00001 /* critical relative x-value difference */\n#define YEPS 0.1 /* critical y-value difference */\n#define EYEPS 0.001 /* critical relative exp(y) difference */\n#define YCEIL 50. /* maximum y avoiding overflow in exp(y) */\n\n/* declarations for functions defined in this file (minus those in arms.h). */\nvoid invert(double prob, arms_state *env, POINT *p);\nint test(arms_state *state, POINT *p, apop_arms_settings *params, gsl_rng *r);\nint update(arms_state *state, POINT *p, apop_arms_settings *params);\nstatic void cumulate(arms_state *env);\nint meet (POINT *q, arms_state *state, apop_arms_settings *params);\ndouble area(POINT *q);\ndouble expshift(double y, double y0);\ndouble logshift(double y, double y0);\ndouble perfunc(apop_arms_settings*, double x);\nvoid display(FILE *f, arms_state *env, apop_arms_settings *);\nint initial (apop_arms_settings* params, arms_state *state);\n\nApop_settings_copy(apop_arms,\n out->state = malloc(sizeof(arms_state));\n *out->state = *in->state;\n)\n\nApop_settings_free(apop_arms,\n if (in->state){\n free(in->state->p);\n free(in->state);\n\t}\n)\n\nApop_settings_init(apop_arms,\n if ((in.xl || in.xr) && !in.xinit)\n out->xinit = (double []) {in.xl+GSL_DBL_EPSILON, (in.xl+in.xr)/2., in.xr-GSL_DBL_EPSILON};\n else{\n //Apop_varad_set(xinit, ((double []) {0, 0.5, 1}));\n Apop_varad_set(xinit, ((double []) {-1, 0, 1}));\n }\n Apop_varad_set(ninit, 3);\n Apop_varad_set(xl, GSL_MIN(out->xinit[0]/10., out->xinit[0]*10)-.1);\n Apop_varad_set(xr, GSL_MAX(out->xinit[out->ninit-1]/10., out->xinit[out->ninit-1]*10)+.1);\n Apop_varad_set(convex, 0);\n Apop_varad_set(npoint, 100);\n Apop_varad_set(do_metro, 'y');\n Apop_varad_set(xprev, (out->xinit[0]+out->xinit[out->ninit-1])/2.);\n Apop_varad_set(neval, 1000);\n Apop_assert(out->model, \"the model input (e.g.: .model = parent_model) is mandatory.\");\n\n // allocate the state \n out->state = malloc(sizeof(arms_state));\n Apop_assert(out->state, \"Malloc failed. Out of memory?\");\n *out->state = (arms_state) { };\n int err = initial(out, out->state);\n Apop_assert_c(!err, NULL, 0, \"init failed, error %i. Returning NULL\", err);\n\n /* finish setting up metropolis struct (can only do this after setting up env) */\n if(out->do_metro=='y'){\n /* I don't understand why this is needed.\n if((params->xprev < params->xl) || (params->xprev > params->xr))\n apop_assert(0, 1007, 0, 's', \"previous Markov chain iterate out of range\")*/\n out->state->metro_xprev = out->xprev;\n out->state->metro_yprev = perfunc(out,out->xprev);\n assert(isfinite(out->state->metro_xprev));\n assert(isfinite(out->state->metro_yprev));\n }\n)\n\nvoid distract_doxygen_arms(){/*Doxygen gets thrown by the settings macros. This decoy function is a workaround. */}\n\n/** Adaptive rejection Metropolis sampling, to make random draws from a univariate distribution.\n\nThe author, Wally Gilks, explains on \nhttp://www.amsta.leeds.ac.uk/~wally.gilks/adaptive.rejection/web_page/Welcome.html , that\n``ARS works by constructing an envelope function of the log of the target density, which is then used in rejection sampling (see, for example, Ripley, 1987). Whenever a point is rejected by ARS, the envelope is updated to correspond more closely to the true log density, thereby reducing the chance of rejecting subsequent points. Fewer ARS rejection steps implies fewer point-evaluations of the log density.''\n\n\\li It accepts only functions with univariate inputs. I.e., it will put a single value into a 1x1 \\ref apop_data set, and then evaluate the log likelihood at that point. For multivariate situations, see \\ref apop_model_metropolis.\n\n\\li It is currently the default for the \\ref apop_draw function given a univariate model, so you can just call that if you prefer.\n\n\\li There are a great number of parameters, in the \\c apop_arms_settings structure. The structure also holds a history of the points tested to date. That means that the system will be more accurate as more draws are made. It also means that if the parameters change, or you use \\ref apop_model_copy, you should call Apop_settings_rm_group(your_model, apop_arms) to clear the model of points that are not valid for a different situation.\n */\nint apop_arms_draw (double *out, gsl_rng *r, apop_model *m){\n apop_arms_settings *params = Apop_settings_get_group(m, apop_arms);\n if (!params) params = Apop_model_add_group(m, apop_arms, .model=m);\n POINT pwork; /* a working point, not yet incorporated in envelope */\n int msamp=0; /* the number of x-values currently sampled */\n arms_state *state = params->state; \n /* now do adaptive rejection */\n do {\n // Sample a new point from piecewise exponential envelope \n double prob = gsl_rng_uniform(r);\n /* get x-value correponding to a cumulative probability prob */\n assert(isfinite(state->p->x));\n assert(isfinite(state->p->y));\n invert(prob,state,&pwork);\n\n /* perform rejection (and perhaps metropolis) tests */\n int i = test(state,&pwork, params, r);\n if (i == 1){ // point accepted \n Apop_notify(3, \" point accepted.\");\n *out = pwork.x;\n assert(isfinite(pwork.x));\n return 0;\n } else \n Apop_stopif(i!=0, return 1,-5, \"envelope error - violation without metropolis\");\n msamp ++;\n Apop_notify(3, \" point rejected.\");\n } while (msamp < 1e3);\n Apop_notify(1, \"I just rejected 1,000 samples. Something is wrong.\");\n return 0;\n}\n\nint initial (apop_arms_settings* params, arms_state *env){\n// to set up initial envelope\n\n POINT *q;\n int mpoint = 2*params->ninit + 1;\n\n Apop_assert_c(params->ninit>=3, 1001, 0, \"too few initial points\");\n Apop_assert_c(params->npoint >= mpoint, 1002, 0, \"too many initial points\");\n Apop_assert_c((params->xinit[0] >= params->xl) && (params->xinit[params->ninit-1] <= params->xr),\n 1003, 0, \"initial points do not satisfy bounds\");\n for(int i=1; ininit; i++)\n Apop_assert_c(params->xinit[i] > params->xinit[i-1], 1004, 0, \"data not ordered\");\n Apop_assert_c(params->convex >= 0.0, 1008, 0, \"negative convexity parameter\");\n\n env->convex = ¶ms->convex; // copy convexity address to env\n params->neval = 0; // initialise current number of function evaluations\n\n /* set up space for envelope POINTs */\n env->npoint = params->npoint;\n env->p = malloc(params->npoint*sizeof(POINT));\n Apop_assert(env->p, \"malloc of env->p failed. Out of space?\");\n\n /* set up envelope POINTs */\n q = env->p;\n q->x = params->xl; // left bound\n q->f = 0;\n q->pl = NULL;\n q->pr = q+1;\n for(int j=1, k=0; jx = params->xinit[k++];\n q->y = perfunc(params,q->x);\n Apop_assert(isfinite(q->x), \"the initial param is %g\", q->x);\n Apop_assert(isfinite(q->y), \"f(an initial parameter)= %g\", q->y);\n q->f = 1;\n } else // intersection point\n q->f = 0;\n q->pl = q-1;\n q->pr = q+1;\n }\n /* right bound */\n q++;\n q->x = params->xr;\n q->f = 0;\n q->pl = q-1;\n q->pr = NULL;\n\n assert(isfinite(q->x));\n /* calculate intersection points */\n q = env->p;\n for (int j=0; jcpoint = mpoint; // note number of POINTs currently in envelope\n return 0;\n}\n\nvoid invert(double prob, arms_state *env, POINT *p){\n/* to obtain a point corresponding to a given cumulative probability \n prob : cumulative probability under envelope \n *env : envelope attributes \n *p : a working POINT to hold the sampled value */\n\n double u,xl=0,xr=0,yl,yr,eyl,eyr,prop;\n\n /* find rightmost point in envelope */\n POINT *q = env->p;\n while(q->pr != NULL)q = q->pr;\n\n /* find exponential piece containing point implied by prob */\n u = prob * q->cum;\n while(q->pl->cum > u)q = q->pl;\n\n /* piece found: set left and right POINTs of p, etc. */\n p->pl = q->pl;\n p->pr = q;\n p->f = 0;\n p->cum = u;\n\n /* calculate proportion of way through integral within this piece */\n prop = (u - q->pl->cum) / (q->cum - q->pl->cum);\n\n /* get the required x-value */\n if (q->pl->x == q->x){\n /* interval is of zero length */\n p->x = q->x;\n p->y = q->y;\n p->ey = q->ey;\n } else {\n xl = q->pl->x;\n xr = q->x;\n yl = q->pl->y;\n yr = q->y;\n eyl = q->pl->ey;\n eyr = q->ey;\n if(fabs(yr - yl) < YEPS){\n /* linear approximation was used in integration in function cumulate */\n if(fabs(eyr - eyl) > EYEPS*fabs(eyr + eyl))\n p->x = xl + ((xr - xl)/(eyr - eyl)) * (-eyl + sqrt((1. - prop)*eyl*eyl + prop*eyr*eyr));\n else \n p->x = xl + (xr - xl)*prop;\n p->ey = ((p->x - xl)/(xr - xl)) * (eyr - eyl) + eyl;\n p->y = logshift(p->ey, env->ymax);\n } else {\n /* piece was integrated exactly in function cumulate */\n p->x = xl + ((xr - xl)/(yr - yl))\n\t * (-yl + logshift(((1.-prop)*eyl + prop*eyr), env->ymax));\n p->y = ((p->x - xl)/(xr - xl)) * (yr - yl) + yl;\n p->ey = expshift(p->y, env->ymax);\n }\n }\n assert(isfinite(p->x));\n assert(isfinite(p->y));\n assert(isfinite(q->x));\n assert(isfinite(q->y));\n\n /* guard against imprecision yielding point outside interval */\n Apop_stopif( ((p->x < xl) || (p->x > xr)), return,-5, \"imprecision yields point outside interval\");\n}\n\nint test(arms_state *env, POINT *p, apop_arms_settings *params, gsl_rng *r){\n/* to perform rejection, squeezing, and metropolis tests \n *env : state data\n *p : point to be tested */\nassert(p->pl && p->pr);\n\n double u,y,ysqueez,ynew,yold,znew,zold,w;\n POINT *ql,*qr;\n \n /* for rejection test */\n u = gsl_rng_uniform(r) * p->ey;\n y = logshift(u,env->ymax);\n\n if(params->do_metro !='y' && (p->pl->pl != NULL) && (p->pr->pr != NULL)){\n /* perform squeezing test */\n ql = p->pl->f ? p->pl : p->pl->pl;\n qr = p->pr->f ? p->pr : p->pr->pr;\n ysqueez = (qr->y * (p->x - ql->x) + ql->y * (qr->x - p->x))\n /(qr->x - ql->x);\n if(y <= ysqueez) // accept point at squeezing step\n return 1;\n }\n\n /* evaluate log density at point to be tested */\n ynew = perfunc(params,p->x);\nassert(isfinite(p->x));\nassert(p->pl && p->pr);\nApop_notify(3, \"tested (%g, %g); \", p->x, ynew);\n \n /* perform rejection test */\n if(params->do_metro != 'y' || (params->do_metro == 'y' && (y >= ynew))){\n /* update envelope */\n p->y = ynew;\n p->ey = expshift(p->y,env->ymax);\n p->f = 1;\n if(update(env,p, params)) \n Apop_assert_c(0, -1, 0, \"envelope violation without metropolis\");\n /* perform rejection test: accept iff y < ynew */\n return (y < ynew);\n }\n\n /* continue with metropolis step */\n yold = env->metro_yprev;\n /* find envelope piece containing metrop->xprev */\n ql = env->p;\n while(ql->pl != NULL) ql = ql->pl;\n while(ql->pr->x < env->metro_xprev) ql = ql->pr;\n qr = ql->pr;\n /* calculate height of envelope at metrop->xprev */\n w = (env->metro_xprev - ql->x)/(qr->x - ql->x);\n zold = ql->y + w*(qr->y - ql->y);\n znew = p->y;\n if(yold < zold)zold = yold;\n if(ynew < znew)znew = ynew;\n w = ynew-znew-yold+zold;\n w = GSL_MIN(w, 0.0);\n w = (w > -YCEIL) ? exp(w) : 0.0;\n u = gsl_rng_uniform(r);\n if(u > w){\n /* metropolis says don't move, so replace current point with previous */\n /* markov chain iterate */\n p->x = env->metro_xprev;\n p->y = env->metro_yprev;\n Apop_notify(3, \"metro step (%g) rejected with w=%g, \"\n \"ynew=%g, yold=%g, znew = %g, zold=%g; \", p->x, w, ynew, yold, znew, zold);\n p->ey = expshift(p->y,env->ymax);\nassert(isfinite(p->x));\nassert(isfinite(p->y));\nassert(isfinite(p->ey));\n p->f = 1;\n p->pl = ql;\n p->pr = qr;\n } else {\n /* trial point accepted by metropolis, so update previous markov */\n /* chain iterate */\n env->metro_xprev = p->x;\n env->metro_yprev = ynew;\n }\n return 1;\n}\n\nint update(arms_state *env, POINT *p, apop_arms_settings *params){\n/* to update envelope to incorporate new point on log density\n *env : state information\n *p : point to be incorporated \n*/\n\n POINT *m,*ql,*qr,*q;\n\n if(!(p->f) || (env->cpoint > env->npoint - 2))\n /* y-value has not been evaluated or no room for further points */\n return 0; // ignore this point\n\n /* copy working POINT p to a new POINT q */\n q = env->p + env->cpoint++;\n q->x = p->x;\n q->y = p->y;\n q->f = 1;\n\n /* allocate an unused POINT for a new intersection */\n m = env->p + env->cpoint++;\n m->f = 0;\n if((p->pl->f) && !(p->pr->f)){\n /* left end of piece is on log density; right end is not */\n /* set up new intersection in interval between p->pl and p */\n m->pl = p->pl;\n m->pr = q;\n q->pl = m;\n q->pr = p->pr;\n m->pl->pr = m;\n q->pr->pl = q;\n } else if (!(p->pl->f) && (p->pr->f)){\n /* left end of interval is not on log density; right end is */\n /* set up new intersection in interval between p and p->pr */\n m->pr = p->pr;\n m->pl = q;\n q->pr = m;\n q->pl = p->pl;\n m->pr->pl = m;\n q->pl->pr = q;\n } else\n Apop_stopif(1, return 1,-5, \"unexpected event\"); // this should be impossible\n\n /* now adjust position of q within interval if too close to an endpoint */\n ql = q->pl->pl ? q->pl->pl : q->pl;\n qr = q->pr->pr ? q->pr->pr : q->pr;\n if (q->x < (1. - XEPS) * ql->x + XEPS * qr->x){\n /* q too close to left end of interval */\n q->x = (1. - XEPS) * ql->x + XEPS * qr->x;\n q->y = perfunc(params,q->x);\n } else if (q->x > XEPS * ql->x + (1. - XEPS) * qr->x){\n /* q too close to right end of interval */\n q->x = XEPS * ql->x + (1. - XEPS) * qr->x;\n q->y = perfunc(params,q->x);\n }\n\n /* revise intersection points */\n if(meet(q->pl,env, params) /* envelope violations without metropolis */\n || meet(q->pr,env, params) \n || (q->pl->pl != NULL && meet(q->pl->pl->pl,env, params))\n || (q->pr->pr != NULL && meet(q->pr->pr->pr,env, params)))\n return 1;\n\n /* exponentiate and integrate new envelope */\n cumulate(env);\n return 0;\n}\n\nstatic void cumulate(arms_state *env){\n/* to exponentiate and integrate envelope */\n/* *env : envelope attributes */\n\n POINT *q,*qlmost;\n\n qlmost = env->p;\n /* find left end of envelope */\n while(qlmost->pl) qlmost = qlmost->pl;\n\n /* find maximum y-value: search envelope */\n env->ymax = qlmost->y;\n for(q = qlmost->pr; q != NULL; q = q->pr)\n if(q->y > env->ymax)\n env->ymax = q->y;\n\n /* exponentiate envelope */\n for(q = qlmost; q != NULL; q = q->pr)\n q->ey = expshift(q->y,env->ymax);\n\n /* integrate exponentiated envelope */\n qlmost->cum = 0.;\n for(q = qlmost->pr; q != NULL; q = q->pr)\n q->cum = q->pl->cum + area(q);\n}\n\nint meet (POINT *q, arms_state *env, apop_arms_settings *params){ \n/* To find where two chords intersect \n q : to store point of intersection \n *env : state attributes \n*/\n double gl=0,gr=0,grl=0,dl=0,dr=0;\n int il=0,ir=0,irl=0;\n\n Apop_assert(!(q->f), \"error 30: this is not an intersection point.\");\n\n /* calculate coordinates of point of intersection */\n if ((q->pl != NULL) && (q->pl->pl->pl != NULL)){\n /* chord gradient can be calculated at left end of interval */\n gl = (q->pl->y - q->pl->pl->pl->y)/(q->pl->x - q->pl->pl->pl->x);\n il = 1;\n } else // no chord gradient on left \n il = 0;\n\n if ((q->pr != NULL) && (q->pr->pr->pr != NULL)){\n /* chord gradient can be calculated at right end of interval */\n gr = (q->pr->y - q->pr->pr->pr->y)/(q->pr->x - q->pr->pr->pr->x);\n ir = 1;\n } else // no chord gradient on right\n ir = 0;\n\n if ((q->pl != NULL) && (q->pr != NULL)){\n /* chord gradient can be calculated across interval */\n grl = (q->pr->y - q->pl->y)/(q->pr->x - q->pl->x);\n irl = 1;\n } else \n irl = 0;\n\n if(irl && il && (gldo_metro !='y') // envelope violation without metropolis\n return 1;\n gl = gl + (1.0 + *(env->convex)) * (grl - gl); // adjust left gradient \n }\n\n if(irl && ir && (gr>grl)){\n /* convexity on right exceeds current threshold */\n if(params->do_metro !='y') // envelope violation without metropolis \n return 1;\n gr = gr + (1.0 + *(env->convex)) * (grl - gr); // adjust right gradient \n }\n\n if(il && irl){\n dr = (gl - grl) * (q->pr->x - q->pl->x);\n if(dr < YEPS) // adjust dr to avoid numerical problems\n dr = YEPS;\n }\n\n if(ir && irl){\n dl = (grl - gr) * (q->pr->x - q->pl->x);\n if(dl < YEPS) // adjust dl to avoid numerical problems \n dl = YEPS;\n }\n\n if(il && ir && irl){\n /* gradients on both sides */\n q->x = (dl * q->pr->x + dr * q->pl->x)/(dl + dr);\n q->y = (dl * q->pr->y + dr * q->pl->y + dl * dr)/(dl + dr);\n } else if (il && irl){\n /* gradient only on left side, but not right hand bound */\n q->x = q->pr->x;\n q->y = q->pr->y + dr;\n } else if (ir && irl){\n /* gradient only on right side, but not left hand bound */\n q->x = q->pl->x;\n q->y = q->pl->y + dl;\n } else if (il)\n q->y = q->pl->y + gl * (q->x - q->pl->x); // right hand bound \n else if (ir)\n q->y = q->pr->y - gr * (q->pr->x - q->x); // left hand bound \n else \n Apop_assert(0, \"error 31: gradient on neither side - should be impossible.\");\n if(((q->pl != NULL) && (q->x < q->pl->x)) ||\n ((q->pr != NULL) && (q->x > q->pr->x))){\n Apop_assert(0, \"error 32: intersection point outside interval (through imprecision)\");\n }\n return 0; // successful exit : intersection has been calculated\n}\n\ndouble area(POINT *q){\n/* To integrate piece of exponentiated envelope to left of POINT q */ \n\n if(q->pl == NULL) // this is leftmost point in envelope\n Apop_stopif(1, return GSL_NAN,-5, \"leftmost point in envelope\");\n if(q->pl->x == q->x) // interval is zero length\n return 0.;\n if (fabs(q->y - q->pl->y) < YEPS) // integrate straight line piece\n return 0.5*(q->ey + q->pl->ey)*(q->x - q->pl->x);\n // integrate exponential piece \n return ((q->ey - q->pl->ey)/(q->y - q->pl->y))*(q->x - q->pl->x);\n}\n\ndouble expshift(double y, double y0) {\n/* to exponentiate shifted y without underflow */\n if (y - y0 > -2.0 * YCEIL)\n return exp(y - y0 + YCEIL);\n else\n return 0.0;\n}\n\ndouble logshift(double y, double y0){\n/* inverse of function expshift */\n return (log(y) + y0 - YCEIL);\n}\n\ndouble perfunc(apop_arms_settings *params, double x){\n// to evaluate log density and increment count of evaluations \n Staticdef( apop_data *, d , apop_data_alloc(1,1));\n d->matrix->data[0] = x;\n double y = apop_log_likelihood(d, params->model);\n Apop_assert(isfinite(y), \"Evaluating the log likelihood at %g returned %g.\", x, y);\n (params->neval)++; // increment count of function evaluations\n return y;\n}\n" }, { "path": "apop_asst.m4.c", "content": "/** \\file apop_asst.c The odds and ends bin. \nCopyright (c) 2005--2007, 2010 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n#include \n#include \n#ifdef _OPENMP\n #include \n #define omp_threadnum omp_get_thread_num()\n#else\n #define omp_threadnum 0\n\n#endif\n\nextern char *apop_nul_string;\n\n//more efficient than xprintf, but a little less versatile.\nstatic void apop_tack_on(char **in, char *addme){\n if (!addme) return;\n size_t inlen = *in? strlen(*in): 0;\n size_t total_len= inlen + strlen(addme);\n *in = realloc(*in, total_len+1);\n strcpy(*in+inlen, addme);\n}\n\ntypedef int (*apop_fn_riip)(apop_data*, int, int, void*);\n\n/** Join together the \\c text grid of an \\ref apop_data set into a single string.\n\nFor example, say that we have a data set with some text: row 0 has\n\\c \"a0\", \\c \"b0\", \\c \"c0\"; row 2 has \n\\c \"a1\", \\c \"b1\", \\c \"c1\"; and so on. We would like to produce\n\n\\code\ninsert into tab values ('a0', 'b0', 'c0');\ninsert into tab values ('a1', 'b1', 'c1');\n...\n\\endcode\n\nThis could be sent to an SQL engine to copy the data to a database (but this is just an example\nfor demonstration---use \\ref apop_data_print to write to a database table).\n\nTo construct this single string from the text grid, we would need to add:\n\\li before the text, Insert into tab values ('.\n\\li between each element on a row: ', '\n\\li between rows: '); \\\\ninsert into tab values('\n\\li at the tail end: ');'\n\nThus, do the conversion via:\n\\code\nchar *insert_string = apop_text_paste(indata,\n .before=\"Insert into tab values ('\",\n .between=\"', '\",\n .between_cols=\"'); \\\\ninsert into tab values(',\n .after=\"');'\"\n);\n\\endcode\n\n\n\\param strings An \\ref apop_data set with a grid of text to be combined into a single string\n\\param between The text to put in between the rows of the table, such as \", \". (Default is a single space: \" \")\n\\param before The text to put at the head of the string. For the query example, this would be .before=\"select \". (Default: NULL)\n\\param after The text to put at the tail of the string. For the query example, .after=\" from data_table\". (Default: NULL)\n\\param between_cols The text to insert between columns of text. See below for an example (Default is set to equal .between)\n\\param prune If you don't want to use the entire text set, you can provide a function to indicate which elements should be pruned out. Some examples:\n\\code\n//Just use column 3\nint is_not_col_3(apop_data *indata, int row, int col, void *ignore){\n return col!=3;\n}\n\n//Jump over blanks as if they don't exist.\nint is_blank(apop_data *indata, int row, int col, void *ignore){\n return strlen(indata->text[row][col])==0;\n}\n\\endcode\n\\param prune_parameter A void pointer to pass to your \\c prune function.\n\n\\return A single string with the elements of the \\c strings table joined as per your\nspecification. Allocated by the function, to be freed by you if desired.\n\n \\li If the table of strings is \\c NULL or has no text, the output string will have\nonly the .before and .after parts with nothing in between.\n \\li if apop_opts.verbose >=3, then print the pasted text to stderr.\n \\li It is sometimes useful to use \\c Apop_r and \\c Apop_rs to get a view of only\none or a few rows in conjunction with this function.\n\n \\li This function uses the \\ref designated syntax for inputs.\n\nThis sample snippet generates the SQL for a query using a list of column names (where\nthe query begins with select , ends with from datatab, and has commas\nin between each element), re-processes the same list to produce the head of an HTML\ntable, then produces the body of the table with the query result.\n\n\\include sql_to_html.c\n*/\nAPOP_VAR_HEAD char *apop_text_paste(apop_data const *strings, char *between, char *before, char *after, char *between_cols, apop_fn_riip prune, void *prune_parameter){\n apop_data const *apop_varad_var(strings, NULL);\n char *apop_varad_var(between, \" \");\n char *apop_varad_var(before, NULL);\n char *apop_varad_var(after, NULL);\n char *apop_varad_var(between_cols, between);\n apop_fn_riip apop_varad_var(prune, NULL);\n void *apop_varad_var(prune_parameter, NULL);\nAPOP_VAR_ENDHEAD\n char *prior_line=NULL, *oneline=NULL, *out = before ? strdup(before) : NULL;\n for (int i=0; i< ((!strings || !*strings->textsize)? 0 : *strings->textsize); i++){\n free(oneline); oneline = NULL;\n for (int j=0; j< strings->textsize[1]; j++){\n if (prune && !prune((apop_data*)strings, i, j, prune_parameter)) continue;\n apop_tack_on(&oneline, strings->text[i][j]);\n if (j textsize[1]-1) apop_tack_on(&oneline, between_cols);\n }\n apop_tack_on(&out, prior_line);\n if (prior_line && oneline) apop_tack_on(&out, between);\n free(prior_line);\n prior_line=oneline ? strdup(oneline): NULL;\n //if (i textsize[0]-1) apop_tack_on(&out, between);\n //if (oneline) apop_tack_on(&out, oneline);\n }\n apop_tack_on(&out, oneline); //the final one never got a chance to be prior_line\n apop_tack_on(&out, after);\n Apop_notify(3, \"%s\", out);\n return out;\n}\n\n/** Calculate \\f$\\sum_{n=1}^N {1\\over n^s}\\f$\n\n\\li There are no doubt efficient shortcuts do doing this, but I use brute force. [Though Knuth's Art of Programming v1 doesn't offer anything, which is strong indication of nonexistence.] To speed things along, I save the results so that they can just be looked up should you request the same calculation. \n\n\\li If \\c N is zero or negative, return NaN. Notify the user if apop_opts.verbosity >=0\n\nFor example: \n\n\\include test_harmonic.c\n*/\nlong double apop_generalized_harmonic(int N, double s){\n/* \nEach row in the saved-results structure is an \\f$s\\f$, and each column is \\f$1\\dots n\\f$, up to the largest \\f$n\\f$ calculated to date.\n\nWhen reading the code, remember that the zeroth element holds the value for N=1, and so on.\n*/\n Apop_stopif(N<=0, return GSL_NAN, 0, \"N is %i, but must be greater than 0.\", N);\n static double * eses\t= NULL;\n static int * \t lengths= NULL;\n static int\t\t count\t= 0;\n static long double ** precalced=NULL;\n int\told_len, i;\n OMP_critical(generalized_harmonic)\n { //Due to memoization, this can't parallelize.\n\tfor (i=0; i< count; i++)\n\t\tif (eses == NULL || eses[i] == s) \t\n break;\n\tif (i == count){\t//you need to build the vector from scratch.\n\t\tcount\t\t\t++;\n i = count - 1;\n\t\tprecalced \t\t= realloc(precalced, sizeof (long double*) * count);\n\t\tlengths \t\t= realloc(lengths, sizeof (int*) * count);\n\t\teses \t\t\t= realloc(eses, sizeof (double) * count);\n\t\tprecalced[i]\t= malloc(sizeof(long double) * N);\n\t\tlengths[i]\t = N;\n\t\teses[i]\t\t = s;\n\t\tprecalced[i][0]\t= 1;\n\t\told_len\t\t\t= 1;\n\t}\n\telse {\t//then you found it.\n\t\told_len = lengths[i];\n\t}\n\tif (N-1 >= old_len){\t//It's there, but you need to extend what you have.\n\t\tprecalced[i] = realloc(precalced[i], sizeof(long double) * N);\n\t\tfor (int j = old_len; jprintf-style arguments. E.g.,\n \n\\code\nchar filenames[] = \"apop_asst.c apop_asst.o\"\napop_system(\"ls -l %s\", filenames);\n\\endcode\n\n\\return The return value of the \\c system() call.\n */\nint apop_system(const char *fmt, ...){\n char *q;\n va_list argp;\n\tva_start(argp, fmt);\n\tApop_stopif(vasprintf(&q, fmt, argp)==-1, return -1, 0, \"Trouble writing to a string.\");\n\tva_end(argp);\n int out = system(q);\n free(q);\n return out;\n}\n\nstatic int count_parens(const char *string){\n int out = 0;\n int last_was_backslash = 0;\n for(const char *step =string; *step !='\\0'; step++){\n if (*step == '\\\\' && !last_was_backslash){\n last_was_backslash = 1;\n continue;\n }\n if (*step == ')' && !last_was_backslash)\n out++;\n last_was_backslash = 0;\n }\n return out;\n}\n\n/** Extract subsets from a string via regular expressions.\n\nThis function takes a regular expression and repeatedly applies it to an input string. It returns the count of matches, and optionally returns the matches themselves organized into the \\c text grid of an \\ref apop_data set.\n\n\\li There are three common flavors of regular expression: Basic, Extended,\nand Perl-compatible (BRE, ERE, PCRE). I use EREs, as per the specs of\nyour C library, which should match POSIX's ERE specification. \n\nFor example, \"p.val\" will match \"P value\", \"p.value\", \"p values\" (and even \"tempeval\", so be\ncareful).\n\nIf you give a non-\\c NULL address in which to place a table of paren-delimited substrings, I'll return them as a row in the text element of the returned \\ref apop_data set. I'll return all the matches, filling the first row with substrings from the first application of your regex, then filling the next row with another set of matches (if any), and so on to the end of the string. Useful when parsing a list of items, for example.\n\n\n\\param string The string to search (no default)\n\\param regex The regular expression (no default)\n\\param substrings Parens in the regex indicate that I should return matching substrings. Give me the _address_ of an \\ref apop_data* set, and I will allocate and fill the text portion with matches. Default= \\c NULL, meaning do not return substrings (even if parens exist in the regex). If no match, return an empty \\ref apop_data set, so output->textsize[0]==0.\n\\param use_case Should I be case sensitive, \\c 'y' or \\c 'n'? (default = \\c 'n', which is not the POSIX default.)\n\n\\return Count of matches found. 0 == no match. \\c substrings may be allocated and filled if needed.\n\n\\li If apop_opts.stop_on_warning='n' returns -1 on error (e.g., regex \\c NULL or didn't compile).\n\\li If strings==NULL, I return 0---no match---and if \\c substrings is provided, set it to \\c NULL.\n\n\\li Here is the test function. Notice that the substring-pulling\nfunction call passes \\c &subs, not plain \\c subs. \n\n\n\\include test_regex.c\n\n\\li Each set of matches will be one row of the output data. E.g., given the regex ([A-Za-z])([0-9]), the column zero of outdata will hold letters, and column one will hold numbers.\nUse \\ref apop_data_transpose to reverse this so that the letters are in outdata->text[0] and numbers in outdata->text[1].\n*/\nAPOP_VAR_HEAD int apop_regex(const char *string, const char* regex, apop_data **substrings, const char use_case){\n const char * apop_varad_var(string, NULL);\n apop_data **apop_varad_var(substrings, NULL);\n if (!string) {\n if (substrings) *substrings=NULL;\n return 0;\n }\n const char * apop_varad_var(regex, NULL);\n Apop_stopif(!regex, return -1, 0, \"You gave me a NULL regex.\");\n const char apop_varad_var(use_case, 'n');\nAPOP_VAR_ENDHEAD\n regex_t re;\n int matchcount=count_parens(regex);\n int found, found_ct=0;\n regmatch_t result[matchcount+1];\n int compiled_ok = !regcomp(&re, regex, REG_EXTENDED \n + (use_case=='y' ? 0 : REG_ICASE)\n + (substrings ? 0 : REG_NOSUB) );\n Apop_stopif(!compiled_ok, return -1, 0, \"This regular expression didn't compile: \\\"%s\\\"\", regex);\n\n int matchrow = 0;\n if (substrings) *substrings = apop_data_alloc();\n do {\n found_ct+=\n found = !regexec(&re, string, matchcount+1, result, matchrow ? REG_NOTBOL : 0);\n if (substrings && found){\n *substrings = apop_text_alloc(*substrings, matchrow+1, matchcount);\n //match zero is the whole string; ignore.\n for (int i=0; i< matchcount; i++){\n if (result[i+1].rm_eo > 0){//GNU peculiarity: match-to-empty marked with -1.\n int length_of_match = result[i+1].rm_eo - result[i+1].rm_so;\n if ((*substrings)->text[matchrow][i] != apop_nul_string) free((*substrings)->text[matchrow][i]);\n (*substrings)->text[matchrow][i] = malloc(strlen(string)+1);\n memcpy((*substrings)->text[matchrow][i], string + result[i+1].rm_so, length_of_match);\n (*substrings)->text[matchrow][i][length_of_match] = '\\0';\n } //else matches nothing; apop_text_alloc already made this cell this NULL.\n }\n string += result[0].rm_eo; //end of whole match;\n matchrow++;\n }\n } while (substrings && found && string[0]!='\\0');\n regfree(&re);\n return found_ct;\n}\n\n/** RNG from a Generalized Hypergeometric type B3.\n\nDevroye uses this as the base for many of his distribution-generators, including the Waring.\n\n\\li If one of the inputs is <=0, error; return NaN and print a warning.\n*/ //Header in stats.h\ndouble apop_rng_GHgB3(gsl_rng * r, double* a){\n Apop_stopif(!((a[0]>0) && (a[1] > 0) && (a[2] > 0)), return NAN, 0, \"all inputs must be positive.\");\n double aa = gsl_ran_gamma(r, a[0], 1),\n\t\t b = gsl_ran_gamma(r, a[1], 1),\n\t\t c = gsl_ran_gamma(r, a[2], 1);\n int\tp = gsl_ran_poisson(r, aa*b/c);\n\treturn p;\n}\n\n/** The Beta distribution is useful for modeling because it is bounded between zero and one, and can be either unimodal (if the variance is low) or bimodal (if the variance is high), and can have either a slant toward the bottom or top of the range (depending on the mean).\n\nThe distribution has two parameters, typically named \\f$\\alpha\\f$ and \\f$\\beta\\f$, which can be difficult to interpret. However, there is a one-to-one mapping between (alpha, beta) pairs and (mean, variance) pairs. Since we have good intuition about the meaning of means and variances, this function takes in a mean and variance, calculates alpha and beta behind the scenes, and returns the appropriate Beta distribution.\n\n\\param m\nThe mean the Beta distribution should have. Notice that m\nis in [0,1].\n\n\\param v\nThe variance which the Beta distribution should have. It is in (0, 1/12), where (1/12) is the variance of a Uniform(0,1) distribution. Funny things happen with variance near 1/12 and mean far from 1/2.\n\n\\return Returns an \\ref apop_model produced by copying the \\c apop_beta model and\nsetting its parameters appropriately.\n\n\\exception out->error=='r' Range error: mean is not within [0, 1].\n*/\napop_model *apop_beta_from_mean_var(double m, double v){\n Apop_stopif(m>=1|| m<=0, apop_model *out = apop_model_copy(apop_beta);\n out->error='r'; return out,\n 0, \"You asked for a beta distribution \"\n \"with mean %g, but the mean of the beta will always \"\n \"be strictly between zero and one.\", m);\n double k = (m * (1- m)/ v) -1;\n double alpha = m*k;\n double beta = k * (1-m);\n return apop_model_set_parameters(apop_beta, alpha, beta);\n}\n\n/** \\def apop_rng_get_thread\nThe \\c gsl_rng is not itself thread-safe, in the sense that it can not be used\nsimultaneously by multiple threads. However, if each thread has its own \\c gsl_rng,\nthen each will safely operate independently.\n\nThus, Apophenia keeps an internal store of RNGs for use by threaded functions. If the\ninput to this function, \\c thread, is greater than any previous input, then the array\nof gsl_rngs is extended to length \\c thread, and each element extended using\n++apop_opts.rng_seed (i.e., the seed is incremented before use).\n\nThis function can be used anywhere a \\c gsl_rng would be used.\n\n\\param thread_in The number of the RNG to retrieve, starting at zero (which is\nhow OpenMP numbers its threads). If -1, I'll look up the current thread (via \\c\nomp_get_thread_num) for you.\n\nSee \\ref threading for additional notes. In most cases, you want to use apop_rng_get_thread(-1).\n\n\\return The appropriate RNG, initialized if necessary.\n\\hideinitializer\n*/\ngsl_rng *apop_rng_get_thread_base(int thread){\n static gsl_rng **rngs;\n static int rng_ct = -1;\n\n if (thread==-1){\n #ifdef OpenMP\n thread = omp_get_thread_num();\n #else\n thread = 0;\n #endif\n }\n\n OMP_critical(rng_get_thread)\n if (thread > rng_ct)\n {\n rngs = realloc(rngs, sizeof(gsl_rng*)*(thread+1));\n for (int i=rng_ct+1; i<= thread; i++)\n rngs[i] = apop_rng_alloc(++apop_opts.rng_seed);\n rng_ct = thread;\n }\n return rngs[thread];\n}\n\n/** Make a set of random draws from a model and write them to an \\ref apop_data set.\n\n\\param model The model from which draws will be made. Must already be prepared and/or estimated.\n\n\\param count The number of draws to make. If \\c draw_matrix is not \\c NULL, then this is ignored and count=draw_matrix->matrix->size1. default=1000.\n\n\\param draws If not \\c NULL, a pre-allocated data set whose \\c matrix element will be filled with draws. \n\n\\return An \\ref apop_data set with the matrix filled with \\c size draws. If draw_matrix!=NULL, then return a pointer to it.\n\n\\exception out->error=='m' Input model isn't good for making draws: it is \\c NULL, or m->dsize=0.\n\n\\exception out->error=='s' You gave me a \\c draws matrix, but its size is less than the size of a single draw from the data, model->dsize.\n\n\\exception out->error=='d' Trouble drawing from the distribution for at least one row. That row is set to all \\c NAN.\n\n\\li Prints a warning if you send in a non-NULL apop_data set, but its \\c matrix element is \\c NULL, when apop_opts.verbose>=1.\n\\li See also \\ref apop_draw, which makes a single draw.\n\\li Random numbers are generated using RNGs from \\ref apop_rng_get_thread, qv.\n\nHere is a two-line program to draw a different set of ten Standard Normals on every run (provided runs are more than a second apart):\n\n\\include draw_some_normals.c\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_model_draws(apop_model *model, int count, apop_data *draws){\n apop_model * apop_varad_var(model, NULL);\n Apop_stopif(!model, apop_return_data_error(n), 0, \"Input model is NULL.\");\n Apop_stopif(!model->dsize, apop_return_data_error(n), 0, \"Input model has dsize==0.\");\n apop_data * apop_varad_var(draws, NULL);\n int apop_varad_var(count, 1000);\n if (draws) {\n Apop_stopif(!draws->matrix, draws->error='m'; return draws, 1, \"Input data set's matrix is NULL.\");\n Apop_stopif((int)draws->matrix->size2 < model->dsize, draws->error='s'; draws->error='m'; return draws,\n 1, \"Input data set's matrix column count is less than model->dsize.\");\n count = draws->matrix->size1;\n } else\n Apop_stopif(model->dsize<=0, apop_return_data_error(n), 0, \"model->dsize<=0, so I don't know the size of matrix to allocate.\");\nAPOP_VAR_ENDHEAD\n apop_data *out = draws ? draws : apop_data_alloc(count, model->dsize);\n\n OMP_for (int i=0; i< count; i++){\n apop_data *onerow = Apop_r(out, i);\n Apop_stopif(apop_draw(onerow->matrix->data, apop_rng_get_thread(omp_threadnum), model),\n gsl_matrix_set_all(onerow->matrix, GSL_NAN); out->error='d',\n 0, \"Trouble drawing for row %i. \"\n \"I set it to all NANs and set out->error='d'.\", i);\n }\n return out;\n}\n" }, { "path": "apop_bootstrap.m4.c", "content": "/** \\file apop_bootstrap.c\n\nCopyright (c) 2006--2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n\n/** Initialize a \\c gsl_rng.\n \n Uses the Tausworth routine.\n\n\\param seed The seed. No need to get funny with it: 0, 1, and 2 will produce wholly different streams.\n\\return The RNG ready for your use.\n\n\\li If you are confident that your code is debugged and would like a new stream of values every time your program runs (provided your runs are more than a second apart), seed with the time:\n\n\\include draw_some_normals.c\n*/\ngsl_rng *apop_rng_alloc(int seed){\n static int first_use = 1;\n if (first_use){\n first_use = 0;\n OMP_critical(rng_env_setup) //GSL makes vague promises about thread-safety\n gsl_rng_env_setup();\n }\n gsl_rng *setme = gsl_rng_alloc(gsl_rng_taus2);\n gsl_rng_set(setme, seed);\n return setme;\n}\n\n/** Give me a data set and a model, and I'll give you the jackknifed covariance matrix of the model parameters.\n\nThe basic algorithm for the jackknife (glossing over the details): create a sequence of data\nsets, each with exactly one observation removed, and then produce a new set of parameter estimates \nusing that slightly shortened data set. Then, find the covariance matrix of the derived parameters.\n\n\\li Jackknife or bootstrap? As a broad rule of thumb, the jackknife works best on models\n that are closer to linear. The worse a linear approximation does (at the given data),\n the worse the jackknife approximates the variance.\n\n\\param in\t The data set. An \\ref apop_data set where each row is a single data point.\n\\param model An \\ref apop_model, that will be used internally by \\ref apop_estimate.\n \n\\exception out->error=='n' \\c NULL input data.\n\\return An \\c apop_data set whose matrix element is the estimated covariance matrix of the parameters.\n\\see apop_bootstrap_cov\n\nFor example:\n\\include jack.c\n*/\napop_data * apop_jackknife_cov(apop_data *in, apop_model *model){\n Apop_stopif(!in, apop_return_data_error(n), 0, \"The data input can't be NULL.\");\n Get_vmsizes(in); //msize1, msize2, vsize\n apop_model *e = apop_model_copy(model);\n int i, n = GSL_MAX(msize1, GSL_MAX(vsize, in->textsize[0]));\n apop_model *overall_est = e->parameters ? e : apop_estimate(in, e);//if not estimated, do so\n gsl_vector *overall_params = apop_data_pack(overall_est->parameters);\n gsl_vector_scale(overall_params, n); //do it just once.\n gsl_vector *pseudoval = gsl_vector_alloc(overall_params->size);\n\n //Copy the original, minus the first row.\n apop_data *subset = apop_data_copy(Apop_rs(in, 1, n-1));\n apop_name *tmpnames = in->names; \n in->names = NULL; //save on some copying below.\n\n apop_data *array_of_boots = apop_data_alloc(n, overall_params->size);\n\n for(i = -1; i< n-1; i++){\n //Get a view of row i, and copy it to position i-1 in the short matrix.\n if (i >= 0) apop_data_memcpy(Apop_r(subset, i), Apop_r(in, i));\n apop_model *est = apop_estimate(subset, e);\n gsl_vector *estp = apop_data_pack(est->parameters);\n gsl_vector_memcpy(pseudoval, overall_params);// *n above.\n gsl_vector_scale(estp, n-1);\n gsl_vector_sub(pseudoval, estp);\n gsl_matrix_set_row(array_of_boots->matrix, i+1, pseudoval);\n apop_model_free(est);\n gsl_vector_free(estp);\n }\n in->names = tmpnames;\n apop_data *out = apop_data_covariance(array_of_boots);\n gsl_matrix_scale(out->matrix, 1./(n-1.));\n apop_data_free(subset);\n gsl_vector_free(pseudoval);\n apop_data_free(array_of_boots);\n if (e!=overall_est)\n apop_model_free(overall_est);\n apop_model_free(e);\n gsl_vector_free(overall_params);\n return out;\n}\n\n/** Give me a data set and a model, and I'll give you the bootstrapped covariance matrix of the parameter estimates.\n\n\\param data\t The data set. An \\c apop_data set where each row is a single data point. (No default)\n\\param model An \\ref apop_model, whose \\c estimate method will be used here. (No default)\n\\param iterations How many bootstrap draws should I make? (default: 1,000) \n\\param rng An RNG that you have initialized, probably with \\c apop_rng_alloc. (Default: an RNG from \\ref apop_rng_get_thread)\n\\param boot_store If not \\c NULL, put the list of drawn parameter values here, with one parameter set per row. Sample use: \n\\code\napop_data *boots;\napop_bootstrap_cov(data, model, .boot_store=&boots);\napop_data_print(boots);\n\\endcode\nThe rows are packed via \\ref apop_data_pack, so use \\ref apop_data_unpack if needed. (Default: \\c NULL)\n\\param ignore_nans If \\c 'y' and any of the elements in the estimation return \\c NaN, then I will throw out that draw and try again. If \\c 'n', then I will write that set of statistics to the list, \\c NaN and all. I keep count of throw-aways; if there are more than \\c iterations elements thrown out, then I throw an error and return with estimates using data I have so far. That is, I assume that \\c NaNs are rare edge cases; if they are as common as good data, you might want to rethink how you are using the bootstrap mechanism. (Default: 'n')\n\\return An \\c apop_data set whose matrix element is the estimated covariance matrix of the parameters.\n\\exception out->error=='n' \\c NULL input data.\n\\exception out->error=='N' \\c too many NaNs.\n\n\\li This function uses the \\ref designated syntax for inputs.\n\nThis example is a sort of demonstration of the Central Limit Theorem. The model is\na simulation, where each call to the estimation routine produces the mean/std dev of\na set of draws from a Uniform Distribution. Because the simulation takes no inputs,\n\\ref apop_bootstrap_cov simply re-runs the simulation and calculates a sequence of\nmean/std dev pairs, and reports the covariance of that generated data set.\n\n\\include boot_clt.c\n\n\\see apop_jackknife_cov\n */\nAPOP_VAR_HEAD apop_data * apop_bootstrap_cov(apop_data * data, apop_model *model, gsl_rng *rng, int iterations, char keep_boots, char ignore_nans, apop_data **boot_store) {\n apop_data * apop_varad_var(data, NULL);\n apop_model *model = varad_in.model;\n int apop_varad_var(iterations, 1000);\n gsl_rng * apop_varad_var(rng, apop_rng_get_thread());\n char apop_varad_var(keep_boots, 'n');\n apop_data** apop_varad_var(boot_store, NULL);\n char apop_varad_var(ignore_nans, 'n');\nAPOP_VAR_ENDHEAD\n Get_vmsizes(data); //vsize, msize1, msize2\n apop_model *e = apop_model_copy(model);\n apop_data *subset = apop_data_copy(data);\n apop_data *array_of_boots = NULL,\n *summary;\n //prevent and infinite regression of covariance calculation.\n Apop_model_add_group(e, apop_parts_wanted); //default wants for nothing.\n size_t i, nan_draws=0;\n apop_name *tmpnames = (data && data->names) ? data->names : NULL; //save on some copying below.\n if (data && data->names) data->names = NULL;\n\n int height = GSL_MAX(msize1, GSL_MAX(vsize, (data?(*data->textsize):0)));\n\tfor (i=0; iparameters);\n if (!gsl_isnan(apop_sum(estp))){\n if (i==0){\n array_of_boots\t = apop_data_alloc(iterations, estp->size);\n apop_name_stack(array_of_boots->names, est->parameters->names, 'c', 'v');\n apop_name_stack(array_of_boots->names, est->parameters->names, 'c', 'c');\n apop_name_stack(array_of_boots->names, est->parameters->names, 'c', 'r');\n }\n gsl_matrix_set_row(array_of_boots->matrix, i, estp);\n } else if (ignore_nans=='y'){\n i--; \n nan_draws++;\n }\n apop_model_free(est);\n gsl_vector_free(estp);\n\t}\n if(data) data->names = tmpnames;\n apop_data_free(subset);\n apop_model_free(e);\n int set_error=0;\n Apop_stopif(i == 0 && nan_draws == iterations, apop_return_data_error(N),\n 1, \"I ran into %i NaNs and no not-NaN estimations, and so stopped. \"\n , iterations);\n Apop_stopif(nan_draws == iterations, set_error++;\n apop_matrix_realloc(array_of_boots->matrix, i, array_of_boots->matrix->size2),\n 1, \"I ran into %i NaNs, and so stopped. Returning results based \"\n \"on %zu bootstrap iterations.\", iterations, i);\n\tsummary\t= apop_data_covariance(array_of_boots);\n if (boot_store) *boot_store = array_of_boots;\n else apop_data_free(array_of_boots);\n if (set_error) summary->error = 'N';\n\treturn summary;\n}\n" }, { "path": "apop_conversions.m4.c", "content": "/** \\file apop_conversions.c\tThe various functions to convert from one format to another. */\n/* Copyright (c) 2006--2010, 2012 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n#include \"apop_internal.h\"\n#include //GSL_NAN\n#include \n#include \n#include \n\n/*extend a string. this prevents a minor leak you'd get if you did\n asprintf(&q, \"%s is a teapot.\", q);\n q may be NULL, which prints the string \"null\", so use the little XN macro below when using this function.\n\n This is internal to apop. right now. \n*/\nvoid xprintf(char **q, char *format, ...){ \n va_list ap; \n char *r = *q; \n va_start(ap, format); \n Apop_stopif(vasprintf(q, format, ap)==-1, , 0, \"Trouble writing to a string.\");\n va_end(ap);\n free(r);\n}\n\n/** Copies a one-dimensional array to a gsl_vector. The input array is undisturbed.\n\n\\param in An array of doubles. (No default. Must not be \\c NULL);\n\\param size \tHow long \\c line is. If this is zero or omitted, I'll\nguess using the sizeof(line)/sizeof(line[0]) trick, which will\nwork for most arrays allocated using double [] and won't work\nfor those allocated using double *. (default = auto-guess)\n\\return A gsl_vector, allocated and filled with a copy of (not a pointer to) the input data.\n\n\\li If you send in a \\c NULL vector, you get a \\c NULL pointer in return. I warn you of this if apop_opts.verbosity >=1 .\n\n\\li This function uses the \\ref designated syntax for inputs.\n\\see \\ref apop_data_falloc\n*/ \nAPOP_VAR_HEAD gsl_vector * apop_array_to_vector(double *in, int size){\n double * apop_varad_var(in, NULL);\n Apop_assert_c(in, NULL, 1, \"You sent me NULL data; returning NULL.\");\n int apop_varad_var(size, sizeof(in)/sizeof(in[0]));\nAPOP_VAR_ENDHEAD\n gsl_vector *out = gsl_vector_alloc(size);\n gsl_vector_view\tv = gsl_vector_view_array((double*)in, size);\n\tgsl_vector_memcpy(out,&(v.vector));\n return out;\n}\n\n/** This function copies the data in a vector to a new one-column (or one-row) matrix\nand returns the newly-allocated and filled matrix.\n\n For the reverse, try \\ref apop_data_pack.\n\n\\param in a \\c gsl_vector (No default. If \\c NULL, I return \\c NULL, with a warning if apop_opts.verbose >=1 )\n\\param row_col If \\c 'r', then this will be a row (1 x N) instead of the default, a column (N x 1). (default: \\c 'c')\n\\return a newly-allocated gsl_matrix with one column (or row).\n\n\\li If you send in a \\c NULL vector, you get a \\c NULL pointer in return. I warn you of this if apop_opts.verbosity >=2 .\n\\li If \\c gsl_matrix_alloc fails you get a \\c NULL pointer in return.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD gsl_matrix * apop_vector_to_matrix(const gsl_vector *in, char row_col){\n const gsl_vector * apop_varad_var(in, NULL);\n Apop_assert_c(in, NULL, 2, \"Converting NULL vector to NULL matrix.\");\n char apop_varad_var(row_col, 'c');\nAPOP_VAR_ENDHEAD\n bool isrow = (row_col == 'r' || row_col == 'R');\n gsl_matrix *out = isrow ? gsl_matrix_alloc(1, in->size)\n : gsl_matrix_alloc(in->size, 1);\n Apop_assert(out, \"gsl_matrix_alloc failed; probably out of memory.\");\n (isrow ? gsl_matrix_set_row\n : gsl_matrix_set_col)(out, 0, in);\n return out;\n}\n\nstatic int find_cat_index(char **d, char * r, int start_from, int size){\n//used for apop_db_to_crosstab.\n int i = start_from % size;\t//i is probably the same or i+1.\n\tdo {\n\t\tif(!strcmp(d[i], r)) return i;\n\t\ti++;\n\t\ti %= size;\t//loop around as necessary.\n\t} while (i!=start_from); \n Apop_assert_c(0, -2, 0, \"Something went wrong in the crosstabbing; couldn't find %s.\", r);\n}\n\n/**Give the name of a table in the database, and optional names of three of its columns:\nthe x-dimension, the y-dimension, and the data. The output is a 2D matrix with rows\nindexed by 'row' and cols by 'col' and the cells filled with the entry in the 'data' column.\n\n\\param tabname The database table I'm querying. Anything that will work inside a \\c from clause is OK, such as a subquery in parens. (no default; must not be \\c NULL)\n\\param row The column of the data set that will indicate the rows of the output crosstab (no default; must not be \\c NULL)\n\\param col The column of the data set that will indicate the columns of the output crosstab (no default; must not be \\c NULL)\n\\param data The column of the data set holding the data for the cells of the crosstab (default: count(*))\n\\param is_aggregate Set to \\c 'y' if the \\c data is a function like count(*)\n or sum(col). That is, set to \\c 'y' if querying this would require a group\n by clause. (default: if I find an end-paren in \\c datacol, \\c 'y'; else \\c 'n'.)\n\n\\li If the query to get data to fill the table (select row, col, data from\n tabname) returns an empty data set, then I will return a \\c NULL data set and if\n apop_opts.verbosity >= 1 print a warning.\n\n\\exception out->error='n' Name not found error.\n\\exception out->error='q' Query returned an empty table (which might mean that it just failed).\n\n\\li The simplest use is to get a tally of how often (r1, r2) appears in the data via apop_db_to_crosstab(\"datatab\", \"r1\", \"r2\").\n\\li If you want a 1-D crosstab, omit the other dimension. Or omit both to get a grand tally of your statistic for the entire table.\n\\li There is a commnad-line tool, apop_db_to_crosstab that calls this function.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_db_to_crosstab(char const*tabname, char const*row, char const* col, char const*data, char is_aggregate){\n char const* apop_varad_var(tabname, NULL);\n Apop_stopif(!tabname, return NULL, 1, \"Missing tabname. Returning NULL.\");\n char const* apop_varad_var(row, \"1\");\n char const* apop_varad_var(col, \"1\");\n char const* apop_varad_var(data, \"count(*)\");\n //This '(' balances the end-paren below, keeping m4 from losing the thread.\n //Note the transitional check for \"group by\", which we should one day remove.\n char apop_varad_var(is_aggregate, (strchr(data, ')') && !strstr(data, \"group by\"))?'y':'n');\nAPOP_VAR_ENDHEAD\n gsl_matrix *out=NULL;\n int\ti, j=0;\n apop_data *pre_d1=NULL, *pre_d2=NULL, *datachars=NULL;\n apop_data *outdata = apop_data_alloc();\n\n char* p = apop_opts.db_name_column;\n apop_opts.db_name_column = NULL;//we put this back at the end.\n char *Q;\n\n Asprintf(&Q, \"select %s, %s, %s from %s %s %s %s %s\", row, col, data, tabname,\n is_aggregate!='n' ? \"group by\" : \"\",\n is_aggregate!='n' ? row : \"\",\n is_aggregate!='n' ? \",\" : \"\",\n is_aggregate!='n' ? col : \"\");\n datachars = apop_query_to_text(\"%s\", Q);\n Apop_stopif(!datachars, free(Q); return NULL, 2, \"[%s] returned an empty table.\", Q);\n Apop_stopif(datachars->error, free(Q); goto bailout, 0, \"error from [%s].\", Q);\n\n //A bit inefficient, but well-encapsulated.\n //Pull the distinct (sorted) list of headers, copy into outdata->names.\n pre_d1 = apop_query_to_text(\"select distinct %s, 1 from %s order by %s\", row, tabname, row);\n Apop_stopif(!pre_d1||pre_d1->error, outdata->error='q'; goto bailout, 0, \"Error querying %s from %s.\", row, tabname);\n for (i=0; i < pre_d1->textsize[0]; i++)\n apop_name_add(outdata->names, pre_d1->text[i][0], 'r');\n\n\tpre_d2 = apop_query_to_text(\"select distinct %s from %s order by %s\", col, tabname, col);\n Apop_stopif(!pre_d2||pre_d2->error, outdata->error='q'; goto bailout, 0, \"Error querying %s from %s.\", row, tabname);\n for (i=0; i < pre_d2->textsize[0]; i++)\n apop_name_add(outdata->names, pre_d2->text[i][0], 'c');\n\n\tout\t= gsl_matrix_calloc(pre_d1->textsize[0], pre_d2->textsize[0]);\n\tfor (size_t k =0; k< datachars->textsize[0]; k++){\n\t\ti = find_cat_index(outdata->names->row, datachars->text[k][0], i, pre_d1->textsize[0]);\n\t\tj = find_cat_index(outdata->names->col, datachars->text[k][1], j, pre_d2->textsize[0]);\n Apop_stopif(i==-2 || j == -2, outdata->error='n'; goto bailout, 0, \"Something went wrong in the crosstabbing; \"\n \"couldn't find %s or %s.\", datachars->text[k][0], datachars->text[k][1]);\n\t\tgsl_matrix_set(out, i, j, atof(datachars->text[k][2]));\n\t}\n bailout:\n apop_data_free(pre_d1);\n apop_data_free(pre_d2);\n apop_data_free(datachars);\n outdata->matrix = out;\n apop_opts.db_name_column = p;\n\treturn outdata;\n}\n\n/** See \\ref apop_db_to_crosstab for the storyline; this is the complement, which takes a\n crosstab and writes its values to the database.\n\nFor example, I would take\n \n\n \n \n
c0c1
r023
r104
\n\nand do the following writes to the database:\n\n\\code\ninsert into your_table values ('r0', 'c0', 2);\ninsert into your_table values ('r0', 'c1', 3);\ninsert into your_table values ('r1', 'c0', 3);\ninsert into your_table values ('r1', 'c1', 4);\n\\endcode\n\n\n\\li If your data set does not have names (or not enough names), I will use the scheme above, filling in names of the form r0, r1, ... c0, c1, .... Text columns get their own names, t0, t1.\n\n\\li This function handles only the matrix and text. \n */\nvoid apop_crosstab_to_db(apop_data *in, char *tabname, char *row_col_name, \n\t\t\t\t\t\tchar *col_col_name, char *data_col_name){\n apop_name *n = in->names;\n char *colname, *rowname;\n Get_vmsizes(in); //msize1, msize2\n int maxcol= GSL_MAX(msize2, in->textsize[1]);\n char sparerow[msize1 > 0 ? (int)log10(msize1)+1 : 0];\n char sparecol[maxcol > 0 ? (int)log10(maxcol)+1 : 0];\n#define DbType apop_opts.db_engine=='m' ? \"text\" : \"character\"\n#define DbType2 apop_opts.db_engine=='m' ? \"double\" : \"numeric\"\n\tapop_query(\"CREATE TABLE %s (%s %s, %s %s, %s %s)\", tabname, \n row_col_name, DbType, col_col_name, DbType, data_col_name, DbType2);\n\tapop_query(\"begin\");\n for (int i=0; i< msize1; i++){\n rowname = (n->rowct > i) ? n->row[i] : (sprintf(sparerow, \"r%i\", i), sparerow);\n for (int j=0; j< msize2; j++){\n colname = (n->colct > j) ? n->col[j] : (sprintf(sparecol, \"c%i\", j), sparecol);\n double x = gsl_matrix_get(in->matrix, i, j); \n if (!isnan(x)) apop_query(\"INSERT INTO %s VALUES ('%s', '%s', %g)\", \n tabname, rowname, colname, x);\n else apop_query(\"INSERT INTO %s VALUES ('%s', '%s', 0/0)\", \n tabname, rowname, colname);\n }\n }\n for (int i=0; i< in->textsize[0]; i++){\n rowname = (n->rowct > i) ? n->row[i] : (sprintf(sparerow, \"r%i\", i), sparerow);\n for (int j=0; j< in->textsize[1]; j++){\n colname = (n->textct > j) ? n->text[j] : (sprintf(sparecol, \"t%i\", j), sparecol);\n apop_query(\"INSERT INTO %s VALUES ('%s', '%s', '%s')\", tabname, \n rowname, colname, in->text[i][j]);\n }\n }\n\tapop_query(\"commit\");\n}\n\n\n/** One often finds data where the column indicates the value of the data point. There may\nbe two columns, and a mark in the first indicates a miss while a mark in the second is a\nhit. Or say that we have the following list of observations:\n\n\\code\n2 3 3 2 1 1 2 1 1 2 1 1\n\\endcode\nThen we could write this as:\n\\code\n0 1 2 3\n----------\n0 6 4 2\n\\endcode\nbecause there are six 1s observed, four 2s observed, and two 3s observed. We call this\nrank format, because 1 (or zero) is typically the most common, 2 is second most common, et cetera.\n\nThis function takes in a list of observations, and aggregates them into a single row in rank format.\n\n\\li For the complement, see \\ref apop_data_rank_expand.\n\n\\li See also \\ref apop_data_to_factors to convert real numbers or text into a\nmatrix of categories.\n\n\\param in The input \\ref apop_data set. If \\c NULL, return \\c NULL.\n\\param min_bins If this is omitted, the number of bins is simply the largest number\nfound. So if there are bins {0, 1, 2} and your data set happens to consist of 0 0\n1 1 0, then I won't know to generate results with three bins where the last bin\nhas a count of zero. Set .min_bins=2 to ensure that bin is included.\n\n\\include test_ranks.c\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data * apop_data_rank_compress (apop_data *in, int min_bins){\n apop_data * apop_varad_var(in, NULL);\n if (!in) return NULL;\n int apop_varad_var(min_bins, 0);\nAPOP_VAR_ENDHEAD\n Get_vmsizes(in);\n int upper_bound = GSL_MAX(in->matrix ? gsl_matrix_max(in->matrix) : min_bins,\n in->vector ? gsl_vector_max(in->vector) : min_bins);\n apop_data *out = apop_data_calloc(1, upper_bound+1);\n for (int i=0; i< msize1; i++)\n for (int j=0; j< msize2; j++) \n (*gsl_matrix_ptr(out->matrix, 0, apop_data_get(in, i, j)))++;\n for (int i=0; i< vsize; i++) \n (*gsl_matrix_ptr(out->matrix, 0, apop_data_get(in, i, -1)))++;\n return out;\n}\n\n/** The complement to this is \\ref apop_data_rank_compress; see that function's\n documentation for the story and an example.\n\n This function takes in a data set where the zeroth column includes the count(s)\n of times that zero was observed, the first gives the count(s) of times that one was\n observed, et cetera. It outputs a data set whose vector element includes a list that\n has exactly the given frequency of zeros, ones, et cetera.\n*/\napop_data *apop_data_rank_expand (apop_data *in){\n int total_ct = (in->matrix ? apop_matrix_sum(in->matrix) : 0)\n + (in->vector ? apop_vector_sum(in->vector) : 0);\n if (total_ct == 0)\n return NULL;\n apop_data *out = apop_data_alloc(total_ct);\n int posn = 0;\n for (int i=0; i< in->matrix->size1; i++)\n for (int k=0; k< in->matrix->size2; k++)\n for (int j=0; j< gsl_matrix_get(in->matrix, i, k); j++)\n gsl_vector_set(out->vector, posn++, k); \n return out;\n}\n\n/** Copy one gsl_vector to another. That is, all data is duplicated.\nUnlike gsl_vector_memcpy, this function allocates and returns the destination,\nso you can use it like this:\n\\code\ngsl_vector *a_copy = apop_vector_copy(original);\n\\endcode\n\n\\param in The input vector\n\\return A structure that this function will allocate and fill. If \\c gsl_vector_alloc fails, returns \\c NULL and print a warning.\n*/\ngsl_vector *apop_vector_copy(const gsl_vector *in){\n if (!in) return NULL;\n gsl_vector *out = gsl_vector_alloc(in->size);\n Apop_stopif(!out, return NULL, 0, \"failed to allocate a gsl_vector of size %zu. Out of memory?\", in->size);\n gsl_vector_memcpy(out, in);\n return out;\n}\n\n/** Copy one gsl_matrix to another. That is, all data are duplicated.\nUnlike gsl_matrix_memcpy, this function allocates and returns the destination,\nso you can use it like this:\n\\code\ngsl_matrix *a_copy = apop_matrix_copy(original);\n\\endcode\n\n\\param in the input data\n\\return A structure that this function will allocate and fill. If \\c gsl_matrix_alloc fails, returns \\c NULL.\n*/\ngsl_matrix *apop_matrix_copy(const gsl_matrix *in){\n if (!in) return NULL;\n gsl_matrix *out = gsl_matrix_alloc(in->size1, in->size2);\n Apop_stopif(!out, return NULL, 0, \"failed to allocate a gsl_matrix of size %zu x %zu. Out of memory?\", in->size1, in->size2);\n gsl_matrix_memcpy(out, in);\n return out;\n}\n\n\n///////////////The text processing section\n\n/** \\page text_format Input text file formatting\n\nThis reference section describes the assumptions made by \\ref apop_text_to_db and \\ref apop_text_to_data.\n\nEach row of the file will be converted to one record in the database or one row in the\nmatrix. Values on one row are separated by delimiters. Fixed-width input is also OK;\nsee below.\n\nBy default, the delimiters are set to \"|,\\t\", meaning that a pipe, comma, or tab\nwill delimit separate entries. To change the default, use an argument to\n\\ref apop_text_to_db or \\ref apop_text_to_data like .delimiters=\" \\t\" or\n.delimiters=\"|\".\n\nThe input text file must be UTF-8 or traditional ASCII encoding. Delimiters must be ASCII characters. \nIf your data is in another encoding, try the POSIX-standard \\c iconv program to filter the data to UTF-8.\n\n \\li The character after a backslash is read as a normal character, even if it is a delimiter, \\c #, or \\c \".\n\\li If a field contains several such special characters, surround it by \\c \"s. The\nsurrounding marks are stripped and the text read verbatim.\n \\li Text does not need to be delimited by quotes (unless there are special characters). If a text field is quote-delimited, I'll strip them.\nE.g., \"Males, 30-40\", is an OK column name, as is \"Males named \\\"Joe\\\\\"\".\n \\li Everything after an unprotected \\c # is taken to be comments and ignored. \n \\li Blank lines (empty or consisting only of white space) are also ignored.\n \\li If you are reading into the gsl_matrix element of an \\ref apop_data set,\nall text fields are taken as zeros. You will be warned of such substitutions unless\nyou set apop_opts.verbose==0 beforehand. For mixed text/numeric data,\ntry using \\ref apop_text_to_db and then \\ref apop_query_to_mixed_data.\n \\li There are often two delimiters in a row, e.g., \"23, 32,, 12\". When it's two commas\nlike this, the user typically means that there is a missing value and the system should\ninsert a NAN; when it is two tabs in a row, this is typically just a formatting\nglitch. Thus, if there are multiple delimiters in a row, I check whether the second\n(and subsequent) is a space or a tab; if it is, then it is ignored, and if it is any\nother delimiter (including the end of the line) then a NaN is inserted.\n\nIf this rule doesn't work for your situation, you can explicitly insert a note that there is a missing data\npoint. E.g., try: \\code\n\t\tperl -pi.bak -e 's/,,/,NaN,/g' data_file\n\\endcode\n\nIf you have missing data delimiters, you will need to set \\ref apop_opts_type\n\"apop_opts.nan_string\" to text that matches the given format. E.g.,\n\n\\code\n//Apophenia's default NaN string, matching NaN, nan, or NAN, but not Nancy:\napop_opts.nan_string = \"NaN\";\n//Popular alternatives:\napop_opts.nan_string = \"Missing\";\napop_opts.nan_string = \".\";\n\n//Or, turn off nan-string checking entirely with:\napop_opts.nan_string = NULL;\n\\endcode\n\nSQLite stores these NaN-type values internally as \\c NULL; that means that functions like\n\\ref apop_query_to_data will convert both your \\c nan_string string and \\c NULL to \\c NaN.\n\n \\li The system uses the standards for C's \\c atof() function for\nfloating-point numbers: INFINITY, -INFINITY, and NaN work as expected.\n \\li If there are row names and column names, then the input will not be perfectly square:\nthere should be no first entry in the sequence of column names like row names. That is,\nfor a 100x100 data set with row and column names, there are 100 names in the top row,\nand 101 entries in each subsequent row (name plus 100 data points).\n \\li White space before or after a field is ignored. So 1, 2,3, 4 , 5, \" six \",7 \nis eqivalent to 1,2,3,4,5,\" six \",7.\n \\li NUL characters ('\\0') are treated as white space, so if your fields have NULs as padding, you should have no problem. NULs inside of a string terminates the string as it always does in C.\n \\li Fixed-width formats are supported (for plain ASCII encoding only), but you have to provide a list of field ending positions. For example, given\n\\code\nNUMLEOL\n123AABB\n456CCDD\n\\endcode\nand .field_ends=(int[]){3, 5, 7}, we have three columns, named NUM, LE,\nand OL. The names can be read from the first row by setting .has_row_names='y'.\n*/\n\nstatic int prep_text_reading(char const *text_file, FILE **infile){\n *infile = !strcmp(text_file, \"-\")\n ? stdin\n\t : fopen(text_file, \"r\");\n Apop_assert_c(*infile, 1, 0, \"Trouble opening %s. Returning NULL.\", text_file);\n return 0;\n}\n\n/////New text file reading\n/** \\cond doxy_ignore */\nextern char *apop_nul_string;\n\n#define Textrealloc(str, len) (str) = \\\n (str) != apop_nul_string \\\n ? realloc((str), (len)) \\\n : (((len) > 0) ? malloc(len) : apop_nul_string);\n\ntypedef struct {int ct; int eof;} line_parse_t;\n/** \\endcond */\n\nstatic line_parse_t parse_a_fixed_line(FILE *infile, apop_data *fn, int const *field_ends){\n int c = fgetc(infile);\n int ct = 0, posn=0, thisflen=0, needfield=1;\n while(c!='\\n' && c !=EOF){\n posn++;\n if (needfield){//start a new field\n if (++ct > fn->textsize[0])\n apop_text_alloc(fn, ct, 1);//realloc text portion.\n thisflen = \n needfield = 0;\n }\n\n //extend field:\n thisflen++;\n Textrealloc(*fn->text[ct-1], thisflen);\n fn->text[ct-1][0][thisflen-1] = c;\n\n if (posn==*field_ends){ //close off this field.\n Textrealloc(*fn->text[ct-1], thisflen+1);\n fn->text[ct-1][0][thisflen] = '\\0';\n thisflen = 0;\n field_ends++;\n needfield=1;\n } \n c = fgetc(infile);\n }\n if (needfield==0){//user didn't give last field end.\n Textrealloc(*fn->text[ct-1], thisflen+1);\n fn->text[ct-1][0][thisflen] = '\\0';\n }\n return (line_parse_t) {.ct=ct, .eof= (c == EOF)};\n}\n\n/** \\cond doxy_ignore */\ntypedef struct{\n char c, type;\n} apop_char_info;\n/** \\endcond */\n\nstatic const size_t bs=1e5;\nstatic int get_next(char *buffer, size_t *ptr, FILE *infile){\n int r;\n if (*ptr>=bs){\n size_t len=fread(buffer, 1, bs, infile);\n if (len < bs) buffer[len]=(char)-1;\n *ptr=0;\n }\n r = buffer[(*ptr)++];\n return r == (char)-1 ? EOF : r;\n}\n\nstatic apop_char_info parse_next_char(char *buffer, size_t *ptr, FILE *f, char const *delimiters){\n int c = get_next(buffer, ptr, f);\n int is_delimiter = !!strchr(delimiters, c);\n return (apop_char_info){.c=c, \n .type = (c==' '||c=='\\r' ||c=='\\t' || c==0)? (is_delimiter ? 'W' : 'w')\n :is_delimiter ? 'd'\n :(c == '\\n') ? 'n'\n :(c == '\"') ? '\"'\n :(c == '\\\\') ? '\\\\'\n :(c == EOF) ? 'E'\n :(c == '#') ? '#'\n : 'r'\n };\n}\n\n//fills fn with a list of strings.\n//returns the count of elements. Negate the count if we're at EOF.\n//fn must already be allocated via apop_data_alloc() [no args].\nstatic line_parse_t parse_a_line(FILE *infile, char *buffer, size_t *ptr, apop_data *fn, int const *field_ends, char const *delimiters){\n int ct=0, thisflen=0, inqq=0, infield=0, mlen=5,\n lastwhite=0, lastnonwhite=0; \n if (field_ends) return parse_a_fixed_line(infile, fn, field_ends);\n apop_char_info ci;\n do {\n ci = parse_next_char(buffer, ptr, infile, delimiters);\n //comments are to end of line, so they're basically a newline.\n if (ci.type=='#' && !inqq){\n for(int c='x'; (c!='\\n' && c!=EOF); )\n c = get_next(buffer, ptr, infile);\n ci.type='n';\n }\n\n //The escape-type cases: \\\\ and \"\".\n //If one applies, set the type to regular\n if (ci.type=='\\\\'){\n ci=parse_next_char(buffer, ptr, infile, delimiters);\n if (ci.type!='E')\n ci.type='r';\n }\n if ((inqq && ci.type !='\"') && ci.type !='E')\n ci.type='r';\n else if (ci.type=='\"') inqq = !inqq;\n\n if (ci.type=='W' && lastwhite==1) \n continue; //compress these.\n lastwhite=(ci.type=='W');\n\n if (!infield){\n if (ci.type=='w') continue; //eat leading spaces.\n if (ci.type=='r' || ci.type=='d' //new field; if 'dnE', blank field. \n || (strchr(\"nE\", ci.type) && ct>0)){ //Blank fields only at end of lines that already have data; else all-blank line to ignore.\n if (++ct > fn->textsize[0]) apop_text_alloc(fn, ct, 1);//realloc text portion.\n Textrealloc(*fn->text[ct-1], 5);\n thisflen = 0;\n mlen=5;\n infield=1;\n } \n } \n if (infield){\n if (ci.type=='d'||ci.type=='n' || ci.type=='E' || ci.type=='W'){\n //delimiter; close off this field.\n fn->text[ct-1][0][lastnonwhite] = '\\0';\n infield =\n thisflen =\n lastnonwhite = 0;\n } else if (ci.type=='w' || ci.type=='r'){ //extend field\n thisflen++; //length of string\n if (thisflen+2 > mlen){\n mlen *=2; //length of allocated memory\n Textrealloc(*fn->text[ct-1], mlen);\n }\n fn->text[ct-1][0][thisflen-1] = ci.c;\n if (ci.type!='w')\n lastnonwhite = thisflen;\n }\n }\n } while (ci.type != 'n' && ci.type != 'E');\n return (line_parse_t) {.ct=ct, .eof= (ci.type == 'E')};\n}\n\n//On return, fn has copies of the field names, and add_this_line has the first data line.\nstatic void get_field_names(int has_col_names, char **field_names, FILE *infile, char *buffer, size_t *ptr,\n apop_data *add_this_line, apop_data *fn, int const *field_ends, char const *delimiters){\n if (has_col_names && field_names == NULL){\n while (fn->textsize[0] ==0) parse_a_line(infile, buffer, ptr, fn, field_ends, delimiters);\n while (add_this_line->textsize[0] ==0) parse_a_line(infile, buffer, ptr, add_this_line, field_ends, delimiters);\n } else{\n while (add_this_line->textsize[0] ==0) \n parse_a_line(infile, buffer, ptr, add_this_line, field_ends, delimiters);\n fn\t= apop_text_alloc(fn, add_this_line->textsize[0], 1);\n for (int i=0; i< fn->textsize[0]; i++)\n if (field_names) apop_text_set(fn, i, 0, field_names[i]);\n else apop_text_set(fn, i, 0, \"col_%i\", i);\n }\n}\n\n/** Read a delimited or fixed-wisdth text file into the matrix element of an \\ref apop_data set.\n\nSee \\ref text_format.\n\nSee also \\ref apop_text_to_db, which handles text data, and may othewise be a perferable approach to data management.\n\n\\param text_file = \"-\" The name of the text file to be read in. If \"-\" (the default), use stdin.\n\\param has_row_names Does the lines of data have row names? \\c 'y' =yes; \\c 'n' =no (default: 'n')\n\\param has_col_names Is the top line a list of column names? See \\ref text_format for notes on dimension (default: 'y')\n\\param field_ends If fields have a fixed size, give the end of each field, e.g. .field_ends=(int[]){3, 8 11}. (default: \\c NULL, indicating not fixed width)\n\\param delimiters A string listing the characters that delimit fields. (default: \"|,\\t\")\n\\return \tReturns an apop_data set.\n\\exception out->error=='a' allocation error\n\\exception out->error=='t' text-reading error\n\nexample: See \\ref apop_ols.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data * apop_text_to_data(char const*text_file, int has_row_names, int has_col_names, int const *field_ends, char const *delimiters){\n char const *apop_varad_var(text_file, \"-\")\n int apop_varad_var(has_row_names, 'n')\n int apop_varad_var(has_col_names, 'y')\n if (has_row_names==1||has_row_names=='Y') has_row_names ='y';\n if (has_col_names==1||has_col_names=='Y') has_col_names ='y';\n int const * apop_varad_var(field_ends, NULL);\n const char * apop_varad_var(delimiters, apop_opts.input_delimiters);\nAPOP_VAR_ENDHEAD\n apop_data *set = NULL;\n FILE *infile = NULL;\n char *str;\n char buffer[bs];\n size_t ptr=bs;\n apop_data *add_this_line= apop_data_alloc();\n int row = 0,\n hasrows = (has_row_names == 'y');\n Apop_stopif(prep_text_reading(text_file, &infile), apop_return_data_error(t),\n 0, \"trouble opening %s\", text_file);\n\n line_parse_t L={ };\n //First, handle the top line, if we're told that it has column names.\n if (has_col_names=='y'){\n apop_data *field_names = apop_data_alloc();\n get_field_names(1, NULL, infile, buffer, &ptr, add_this_line, field_names, field_ends, delimiters);\n L.ct = *add_this_line->textsize;\n set = apop_data_alloc(0,1, L.ct - hasrows);\n\t set->names->colct = 0;\n\t set->names->col = malloc(sizeof(char*));\n for (int j=0; j< L.ct - hasrows; j++)\n apop_name_add(set->names, *field_names->text[j], 'c');\n apop_data_free(field_names);\n } \n\n //Now do the body.\n\twhile(!set || !L.eof || L.ct){\n if (!L.ct) { //skip blank lines\n L=parse_a_line(infile,buffer, &ptr, add_this_line, field_ends, delimiters);\n continue;\n }\n if (!set) set = apop_data_alloc(0, 1, L.ct-hasrows); //for .has_col_names=='n'.\n row++;\n int cols = set->matrix ? set->matrix->size2 : L.ct - hasrows;\n set->matrix = apop_matrix_realloc(set->matrix, row, cols);\n Apop_stopif(!set->matrix, set->error='a'; return set, 0, \"allocation error.\");\n if (hasrows) {\n apop_name_add(set->names, *add_this_line->text[0], 'r');\n Apop_stopif(L.ct-1 > set->matrix->size2, set->error='t'; return set, 1,\n \"row %i (not counting rownames) has %i elements (not counting the rowname), \"\n \"but I thought this was a data set with %zu elements per row. \"\n \"Stopping the file read; returning what I have so far.\", row, L.ct-1, set->matrix->size2);\n } else Apop_stopif(L.ct > set->matrix->size2, set->error='t'; return set, 1,\n \"row %i has %i elements, \"\n \"but I thought this was a data set with %zu elements per row. \"\n \"Stopping the file read; returning what I have so far. Set has_row_names?\", row, L.ct, set->matrix->size2);\n for (int col=hasrows; col < L.ct; col++){\n char *thisstr = *add_this_line->text[col];\n if (strlen(thisstr)){\n double val = strtod(thisstr, &str);\n if (thisstr != str)\n gsl_matrix_set(set->matrix, row-1, col-hasrows, val);\n else {\n gsl_matrix_set(set->matrix, row-1, col-hasrows, GSL_NAN);\n Apop_notify(1, \"trouble converting data item %i on data line %i [%s]; writing NaN.\", col, row, thisstr);\n }\n } else gsl_matrix_set(set->matrix, row-1, col-hasrows, GSL_NAN);\n }\n if (L.eof) break;//hit when the last line has elements and is terminated by EOF.\n L=parse_a_line(infile, buffer, &ptr, add_this_line, field_ends, delimiters);\n\t}\n apop_data_free(add_this_line);\n if (strcmp(text_file,\"-\")) fclose(infile);\n\treturn set;\n}\n\n/** This is the complement to \\ref apop_data_pack, qv. It writes the \\c gsl_vector\n produced by that function back to the \\ref apop_data set you provide. It overwrites\n the data in the vector and matrix elements and, if present, the \\c weights (and\n that's it, so names or text are as before).\n\n\\param in A \\c gsl_vector of the form produced by \\ref apop_data_pack. No default; must not be \\c NULL.\n\\param d That data set to be filled. Must be allocated to the correct size. No default; must not be \\c NULL.\n\\param use_info_pages Pages in XML-style brackets, such as \\ will\nbe ignored unless you set .use_info_pages='y'. Be sure that this is set to the\nsame thing when you both pack and unpack. (Default: \\c 'n').\n\n\\li If I get to the end of the first page of the \\c apop_data set and have more\n entries in the vector to unpack, and the data to fill has a \\c more element,\n then I will continue into subsequent pages.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD void apop_data_unpack(const gsl_vector *in, apop_data *d, char use_info_pages){\n const gsl_vector * apop_varad_var(in, NULL);\n apop_data* apop_varad_var(d, NULL);\n Apop_stopif(!d, return, 0, \"the data set to be filled, d, must not be NULL\");\n char apop_varad_var(use_info_pages, 'n');\nAPOP_VAR_ENDHEAD\n int offset = 0;\n gsl_vector vin, vout;\n if(d->vector){\n vin = gsl_vector_subvector((gsl_vector *)in, 0, d->vector->size).vector;\n gsl_vector_memcpy(d->vector, &vin);\n offset += d->vector->size;\n }\n if(d->matrix)\n for (size_t i=0; i< d->matrix->size1; i++){\n vin = gsl_vector_subvector((gsl_vector *)in, offset, d->matrix->size2).vector;\n vout = gsl_matrix_row(d->matrix, i).vector;\n gsl_vector_memcpy(&vout, &vin);\n offset += d->matrix->size2;\n }\n if(d->weights){\n vin = gsl_vector_subvector((gsl_vector *)in, offset, d->weights->size).vector;\n gsl_vector_memcpy(d->weights, &vin);\n offset += d->weights->size;\n }\n if (offset != in->size && d->more){\n vin = gsl_vector_subvector((gsl_vector *)in, offset, in->size - offset).vector;\n d = d->more;\n if (use_info_pages=='n')\n while (d && apop_regex(d->names->title, \"^<.*>$\"))\n d = d->more;\n Apop_stopif(!d, return, 0, \"The data set (without info pages, because you didn't ask\"\n \" me to use them) is too short for the input vector.\");\n apop_data_unpack(&vin, d);\n }\n}\n\nstatic size_t sizecount(const apop_data *in, bool all_pp, bool use_info_pp){ \n if (!in) return 0;\n if (!use_info_pp && in->names && apop_regex(in->names->title, \"^<.*>$\"))\n return (all_pp ? sizecount(in->more, all_pp, use_info_pp) : 0);\n return (in->vector ? in->vector->size : 0)\n + (in->matrix ? in->matrix->size1 * in->matrix->size2 : 0)\n + (in->weights ? in->weights->size : 0)\n + (all_pp ? sizecount(in->more, all_pp, use_info_pp) : 0);\n}\n\n/** This function takes in an \\ref apop_data set and writes it as a single column of\nnumbers, outputting a \\c gsl_vector.\n It is valid to use the \\c out_vector->data element as an array of \\c doubles of size\n \\c out_vector->data->size (i.e. its stride==1).\n\n The complement is \\c apop_data_unpack. I.e., \n\\code\napop_data_unpack(apop_data_pack(in_data), data_copy) \n\\endcode\nwill return the original data set (stripped of text and names).\n\n \\param in an \\c apop_data set. No default; if \\c NULL, return \\c NULL.\n \\param out If this is not \\c NULL, then put the output here. The dimensions must match exactly. If \\c NULL, then allocate a new data set. Default = \\c NULL. \n \\param more_pages If \\c 'y', then follow the ->more pointer to fill subsequent\npages; else fill only the first page. Informational pages will still be ignored, unless you set .use_info_pages='y' as well. Default = \\c 'y'. \n\\param use_info_pages Pages in XML-style brackets, such as \\ will\nbe ignored unless you set .use_info_pages='y'. Be sure that this is set to the\nsame thing when you both pack and unpack. Default: 'n'.\n\n \\return A \\c gsl_vector with the vector data (if any), then each row of data (if any), then the weights (if any), then the same for subsequent pages (if any && .more_pages=='y'). If \\c out is not \\c NULL, then this is \\c out.\n\\exception NULL If you give me a vector as input, and its size is not correct, returns \\c NULL.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD gsl_vector * apop_data_pack(const apop_data *in, gsl_vector *out, char more_pages, char use_info_pages){\n const apop_data * apop_varad_var(in, NULL);\n if (!in) return NULL;\n gsl_vector * apop_varad_var(out, NULL);\n char apop_varad_var(more_pages, 'y');\n char apop_varad_var(use_info_pages, 'n');\n if (out) {\n size_t total_size = sizecount(in, (more_pages == 'y' || more_pages == 'Y'), (use_info_pages =='y' || use_info_pages =='Y'));\n Apop_stopif(out->size != total_size, return NULL, 0, \"The input data set has %zu elements, \"\n \"but the output vector you want to fill has size %zu. Please make \"\n \"these sizes equal.\", total_size, out->size);\n }\nAPOP_VAR_ENDHEAD\n size_t total_size = sizecount(in, (more_pages == 'y' || more_pages == 'Y'), (use_info_pages =='y' || use_info_pages =='Y'));\n if (!total_size) return NULL;\n int offset = 0;\n if (!out) out = gsl_vector_alloc(total_size);\n gsl_vector vout, vin;\n if (in->vector){\n vout = gsl_vector_subvector((gsl_vector *)out, 0, in->vector->size).vector;\n gsl_vector_memcpy(&vout, in->vector);\n offset += in->vector->size;\n }\n if (in->matrix)\n for (size_t i=0; i< in->matrix->size1; i++){\n vin = gsl_matrix_row(in->matrix, i).vector;\n vout= gsl_vector_subvector((gsl_vector *)out, offset, in->matrix->size2).vector;\n gsl_vector_memcpy(&vout, &vin);\n offset += in->matrix->size2;\n }\n if (in->weights){\n vout = gsl_vector_subvector((gsl_vector *)out, offset, in->weights->size).vector;\n gsl_vector_memcpy(&vout, in->weights);\n offset += in->weights->size;\n }\n if ((more_pages == 'y' ||more_pages =='Y') && in->more){\n while (use_info_pages=='n' && in->more && apop_regex(in->more->names->title, \"^<.*>$\"))\n in = in->more;\n if (in->more){\n vout = gsl_vector_subvector((gsl_vector *)out, offset, out->size - offset).vector;\n apop_data_pack(in->more, &vout, more_pages, use_info_pages);\n }\n }\n return out;\n}\n\n/** \\def apop_data_falloc\nAllocate a data set and fill it with values. Put the data set dimensions (one, two,\nor three dimensions as per \\ref apop_data_alloc) in parens, then the data (as per \\ref\napop_data_fill). E.g.:\n\\code\napop_data *identity2 = apop_data_falloc((2,2),\n 1, 0,\n 0, 1);\n\napop_data *count_vector = apop_data_falloc((5), 0, 1, 2, 3, 4);\n\\endcode\n\nIf you forget the parens, you will get an obscure error during compilation.\n\n\\li This is a simple macro wrapping \\ref apop_data_fill and \\ref apop_data_alloc,\nbecause they appear together so often. The second example expands to:\n\\code\napop_data *count_vector = apop_data_fill(apop_data_alloc(5), 0, 1, 2, 3, 4);\n\\endcode\n*/\n\n/** \\def apop_data_fill\nFill a pre-allocated data set with values.\n\n\\param adfin An \\c apop_data set (that you have already allocated).\n\\param ... A series of at least as many floating-point values as there are blanks in the data set.\n\\return A pointer to the same data set that was input.\n\n\\li I need as many arguments as the size of the data set, and can't count them for\nyou. Too many will be ignored; too few will produce unpredictable results, which may\ninclude padding your matrix with garbage or a simple segfault.\n\n\\li Underlying this function is a base function that takes a single list, as opposed\nto the set of unassociated numbers sent to \\ref apop_data_fill. See the example below for a comparison.\n\n\\li This function assumes that if the \\ref apop_data set has both \\c vector and \\c\nmatrix, then vector->size==matrix->size1.\n\n\\li See also \\ref apop_data_falloc to allocate and fill on one line. E.g., to\ngenerate a unit vector for three dimensions:\n\\code\napop_data *unit_vector = apop_data_falloc((3), 1, 1, 1);\n\\endcode\n\nAn example, using both a loose list of numbers and an array.\n\n\\include data_fill.c\n\n\\see apop_text_fill, apop_data_falloc, apop_data_unpack\n*/\n\napop_data *apop_data_fill_base(apop_data *in, double ap[]){\n/* In conversions.h, you'll find this header, which turns all but the first input into an array of doubles of indeterminate length:\n#define apop_data_fill(in, ...) apop_data_fill_base((in), (double []) {__VA_ARGS__})\n*/\n if (!in) return NULL;\n int k=0, start=0, fin=0, height=0;\n if (in->vector){\n start = -1;\n height = in->vector->size;\n }\n if (in->matrix){\n fin = in->matrix->size2;\n height = in->matrix->size1;\n }\n for (int i=0; i< height; i++)\n for (int j=start; j< fin; j++)\n apop_data_set(in, i, j, ap[k++]);\n return in;\n}\n\n/** \\def apop_vector_fill\n Fill a pre-allocated \\c gsl_vector with values.\n\n See \\ref apop_data_alloc for a relevant example. See also \\ref apop_matrix_alloc.\n\nWarning: I need as many arguments as the size of the vector, and can't count them for you. Too many will be ignored; too few will produce unpredictable results, which may include padding your vector with garbage or a simple segfault.\n\n\n\\param avfin A \\c gsl_vector (that you have already allocated).\n\\param ... A series of exactly as many values as there are spaces in the vector.\n\\return A pointer to the same vector that was input.\n*/\ngsl_vector *apop_vector_fill_base(gsl_vector *in, double ap[]){\n if (!in) return NULL;\n for (int i=0; i< in->size; i++)\n gsl_vector_set(in, i, ap[i]);\n return in;\n}\n\n/** \\def apop_text_fill(in, ap)\nFill the text part of an already-allocated \\ref apop_data set with a list of strings. \n\n\\param dataset A data set that you already prepared with \\ref apop_text_alloc.\n\\param ... A list of strings. The first row is filled first, then the second, and so on to the end of the text grid.\n\n\\li If an element is \\c NULL, write apop_opts.nan_string at that point. You may prefer to use \"\" to express a blank.\n\\li If you provide more or fewer strings than are needed to fill the text grid and\n apop_opts.verbose >=1, I print a warning and continue to \n the end of the text grid or data set, whichever is shorter.\n\\li If the data set is \\c NULL, I return \\c NULL. If you provide a \\c NULL data set\n but a non-NULL list of text elements, and apop_opts.verbose >=1, I print\n a warning and return \\c NULL.\n\\li Remember that the C preprocessor concatenates two adjacent strings into one. Here\n is an attempt to fill a \\f$ 2\\times 3\\f$ grid:\n\\code\n apop_data *one23 = apop_text_fill(apop_text_alloc(NULL, 2, 3),\n \"one\", \"two\", \"three\" //missing comma!\n \"two\", \"four\", \"six\");\n\\endcode\nThe preprocessor will join \"three\" \"two\" to form \"threetwo\", leaving you with only five strings.\n\n\\li If you have a \\c NULL-delimited array of strings (not just a loose list as above),\nthen use \\c apop_text_fill_base. \n*/\napop_data *apop_text_fill_base(apop_data *data, char* text[]){\n int textct = 0;\n for (char **textptr = text; *textptr; textptr++) textct++;\n Apop_stopif(!data && textct, return NULL, 1, \"NULL data set input; returning NULL.\");\n if (!data) return NULL;\n int gridsize = data ? data->textsize[0]*data->textsize[1] : 0;\n Apop_stopif(textct != gridsize, /*continue*/, 1, \"Data set has a text grid \"\n \"of size %i but you gave me %i strings.\", gridsize, textct);\n\n int ctr=0;\n for (int i=0; i< data->textsize[0]; i++)\n for (int j=0; j< data->textsize[1]; j++)\n apop_text_set(data, i, j, text[ctr++]);\n return data;\n}\n\n\n///////The rest of this file is for apop_text_to_db\nextern sqlite3 *db;\n\nstatic char *get_field_conditions(char *var, apop_data *field_params){\n if (field_params)\n for (int i=0; itextsize[0]; i++)\n if (apop_regex(var, field_params->text[i][0]))\n return field_params->text[i][1];\n return (apop_opts.db_engine == 'm') ? \"varchar(100)\" : \"numeric\";\n}\n\nstatic int tab_create_mysql(char *tabname, int has_row_names, apop_data *field_params, char *table_params, apop_data const *fn){\n char *q = NULL;\n Asprintf(&q, \"create table %s\", tabname);\n for (int i=0; i < *fn->textsize; i++){\n if (i==0)\n xprintf(&q, has_row_names ? \"%s (row_names varchar(100), \" : \"%s (\", q);\n else xprintf(&q, \"%s %s, \", q, get_field_conditions(*fn->text[i-1], field_params));\n xprintf(&q, \"%s %s\", q, *fn->text[i]);\n }\n xprintf(&q, \"%s %s%s%s)\", q, get_field_conditions(*fn->text[fn->textsize[0]-1], field_params)\n , table_params? \", \": \"\", XN(table_params));\n apop_query(\"%s\", q);\n Apop_stopif(!apop_table_exists(tabname), return -1, 0, \"query \\\"%s\\\" failed.\", q);\n free(q);\n return 0;\n}\n\nstatic int tab_create_sqlite(char *tabname, int has_row_names, apop_data *field_params, char *table_params, apop_data const *fn){\n char *q = NULL;\n Asprintf(&q, \"create table %s\", tabname);\n for (int i=0; itextsize[0]; i++){\n if (i==0){\n if (has_row_names) xprintf(&q, \"%s ('row_names', \", q);\n else xprintf(&q, \"%s (\", q);\n } else xprintf(&q, \"%s' %s, \", q, get_field_conditions(*fn->text[i-1], field_params));\n xprintf(&q, \"%s '%s\", q, *fn->text[i]);\n }\n xprintf(&q, \"%s' %s%s%s);\", q, get_field_conditions(*fn->text[fn->textsize[0]-1], field_params)\n , table_params? \", \": \"\", XN(table_params));\n apop_query(\"%s\", q);\n Apop_stopif(!apop_table_exists(tabname), return -1, 0, \"query \\\"%s\\\" failed.\", q);\n free(q);\n return 0;\n}\n\n/**\n--If the string has zero length, then it's probably a missing value.\n --If the string isn't a number, it needs quotes\n */\nchar *prep_string_for_sqlite(int prepped_statements, char const *astring){\n if (!astring || astring[0]=='\\0' || \n (apop_opts.nan_string && !strcasecmp(apop_opts.nan_string, astring)))\n return NULL;\n\n char *out = NULL,\n\t\t *tail = NULL;\n\tif(strtod(astring, &tail)) \n /*do nothing.*/;\n if (*tail!='\\0'){\t//then it's not a number.\n if (!prepped_statements){\n if (strchr(astring, '\\''))\n Asprintf(&out,\"\\\"%s\\\"\", astring);\n else\n Asprintf(&out,\"'%s'\", astring);\n } else out = strdup(astring);\n\t} else {\t //number, maybe INF or NAN. Also, sqlite wants 0.1, not .1\n\t\tassert(*astring!='\\0');\n if (isinf(atof(astring))==1)\n\t\t\tout = strdup(\"9e9999999\");\n else if (isinf(atof(astring))==-1)\n\t\t\tout = strdup(\"-9e9999999\");\n else if (gsl_isnan(atof(astring)))\n\t\t\tout = strdup(\"0.0/0.0\");\n else if (astring[0]=='.')\n\t\t\tAsprintf(&out, \"0%s\",astring);\n\t\telse out = strdup(astring);\n\t}\n return out;\n}\n\nstatic void line_to_insert(line_parse_t L, apop_data const*addme, char const *tabname, \n sqlite3_stmt *p_stmt, int row){\n if (!L.ct) return;\n int field = 1;\n char comma = ' ';\n char *q = NULL;\n if (!p_stmt) Asprintf(&q, \"INSERT INTO %s VALUES (\", tabname);\n for (int col=0; col < L.ct; col++){\n char *prepped = prep_string_for_sqlite(!!p_stmt, *addme->text[col]);\n if (p_stmt){\n if (!prepped || !strlen(prepped))\n field++; //leave NULL and cleared\n else \n Apop_stopif(sqlite3_bind_text(p_stmt, field++, prepped, -1, SQLITE_TRANSIENT)!=SQLITE_OK,\n /*keep going */, 0, \"Something wrong on line %i, field %i [%s].\\n\"\n , row, field-1, *addme->text[col]);\n } else {\n xprintf(&q, \"%s%c %s\", q, comma, (prepped && strlen(prepped) ? prepped : \" NULL\"));\n comma = ',';\n }\n free(prepped);\n }\n if (!p_stmt){\n apop_query(\"%s)\",q); \n free (q);\n }\n}\n\nint apop_use_sqlite_prepared_statements(size_t col_ct){\n #if SQLITE_VERSION_NUMBER < 3003009\n return 0;\n #else\n return (sqlite3_libversion_number() >=3003009\n && !(apop_opts.db_engine == 'm')\n && col_ct <= 999); //Arbitrary SQLite limit on blanks in prepared statements.\n #endif\n}\n\nint apop_prepare_prepared_statements(char const *tabname, size_t col_ct, sqlite3_stmt **statement){\n #if SQLITE_VERSION_NUMBER < 3003009\n Apop_stopif(1, return -1, 0, \"Attempting to prepapre prepared statements, but using a version of SQLite that doesn't support them.\");\n #else\n char *q=NULL;\n Asprintf(&q, \"INSERT INTO %s VALUES (\", tabname);\n for (size_t i = 0; i < col_ct; i++)\n xprintf(&q, \"%s?%c\", q, i==col_ct-1 ? ')' : ',');\n Apop_stopif(!db, return -1, 0, \"The database should be open by now but isn't.\");\n Apop_stopif(sqlite3_prepare_v2(db, q, -1, statement, NULL) != SQLITE_OK, \n return -1, apop_errorlevel, \"Failure preparing prepared statement: %s\", sqlite3_errmsg(db));\n free(q);\n return 0;\n #endif\n}\n\nstatic char *cut_at_dot(char const *infile){\n char *incopy = strdup(infile); //basename reserves the right to modify its input.\n char *out = strdup(basename(incopy));\n free(incopy);\n char *dot = strchr(out, '.');\n if (dot) *dot='\\0';\n return out;\n}\n\n/** Read a delimited or fixed-width text file into a database table.\n See \\ref text_format. \n\nFor purely numeric data, you may be able to bypass the database by using \\ref apop_text_to_data.\n\nSee the \\ref apop_ols page for an example that uses this function to read in sample data (also listed on that page).\n\nApophenia ships with an \\c apop_text_to_db command-line utility, which is a wrapper for this function.\n\nEspecially if you are using a pre-2007 version of SQLite, there may be a speedup to putting this function in a begin/commit wrapper:\n\\code\napop_query(\"begin;\");\napop_data_print(dataset, .output_name=\"dbtab\", .output_type='d');\napop_query(\"commit;\");\n\\endcode\n\n\\param text_file The name of the text file to be read in. If \\c \"-\", then read from \\c STDIN. (default: \"-\")\n\\param tabname The name to give the table in the database\n (default: \\c text_file after the last slash and up to the next dot. E.g.,\n text_file==\"../data/pant_lengths.csv\" gives tabname==\"pant_lengths\")\n\\param has_row_names Does the lines of data have row names? (default: 0)\n\\param has_col_names Is the top line a list of column names? (default: 1)\n\\param field_names The list of field names, which will be the columns for the table. If has_col_names==1, read the names from the file (and just set this to NULL). If has_col_names == 1 && field_names !=NULL, I'll use the field names. (default: NULL)\n\\param field_ends If fields have a fixed size, give the end of each field, e.g. .field_ends=(int[]){3, 8 11}. (default: \\c NULL, indicating not fixed width)\n\\param field_params There is an implicit create table in setting up the database. If you want to add a type, constraint, or key, put that here. The relevant part of the input \\ref apop_data set is the \\c text grid, which should be \\f$N \\times 2\\f$. The first item in each row (your_params->text[n][0], for each \\f$n\\f$) is a regular expression to match against the variable names; the second item (your_params->text[n][1]) is the type, constraint, and/or key (i.e., what comes after the name in the \\c create query). Not all variables need be mentioned; the default type if nothing matches is numeric. I go in order until I find a regex that matches the given field, so if you don't like the default, then set the last row to have name .*, which is a regex guaranteed to match anything that wasn't matched by an earlier row, and then set the associated type to your preferred default. See \\ref apop_regex on details of matching. (default: NULL)\n\\param table_params There is an implicit create table in setting up the database. If you want to add a table constraint or key, such as not null primary key (age, sex), put that here.\n\\param delimiters A string listing the characters that delimit fields. default = \"|,\\t\"\n\\param if_table_exists What should I do if the table exists?
\n\\c 'n' Do nothing; exit this function. (default)
\n\\c 'd' Retain the table but delete all data; refill with the new data (i.e., call \"delete * from your_table\").
\n\\c 'o' Overwrite the table from scratch; deleting the previous table entirely.
\n\\c 'a' Append new data to the existing table.\n\n\\return Returns the number of rows on success, -1 on error.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_text_to_db(char const *text_file, char *tabname, int has_row_names, int has_col_names, char **field_names, int const *field_ends, apop_data *field_params, char *table_params, char const *delimiters, char if_table_exists){\n char const *apop_varad_var(text_file, \"-\")\n char *apop_varad_var(tabname, cut_at_dot(text_file))\n int apop_varad_var(has_row_names, 'n')\n int apop_varad_var(has_col_names, 'y')\n if (has_row_names==1||has_row_names=='Y') has_row_names ='y';\n if (has_col_names==1||has_col_names=='Y') has_col_names ='y';\n int const *apop_varad_var(field_ends, NULL)\n char ** apop_varad_var(field_names, NULL)\n apop_data * apop_varad_var(field_params, NULL)\n char * apop_varad_var(table_params, NULL)\n const char * apop_varad_var(delimiters, apop_opts.input_delimiters);\n char apop_varad_var(if_table_exists, 'n')\nAPOP_VAR_ENDHEAD\n int batch_size = 10000,\n \t col_ct, ct = 0, rows = 1;\n FILE *infile;\n char buffer[bs];\n size_t ptr = bs;\n apop_data *add_this_line = apop_data_alloc();\n sqlite3_stmt *statement = NULL;\n line_parse_t L = {1,0};\n \n bool tab_exists = apop_table_exists(tabname);\n if (tab_exists){\n Apop_stopif(if_table_exists=='n', return -1, 0, \"table %s exists; not recreating it.\", tabname);\n if (if_table_exists=='d') \n apop_query(\"delete from %s\", tabname);\n else if (if_table_exists=='o') {\n apop_query(\"drop table %s\", tabname); \n tab_exists=false;\n }\n }\n\n //get names and the first row.\n if (prep_text_reading(text_file, &infile)) return -1;\n apop_data *fn = apop_data_alloc();\n get_field_names(has_col_names=='y', field_names, infile, buffer, &ptr,\n add_this_line, fn, field_ends, delimiters);\n col_ct = L.ct = *add_this_line->textsize;\n Apop_stopif(!col_ct, return -1, 0, \"counted zero columns in the input file (%s).\", tabname);\n if (!tab_exists)\n Apop_stopif( ((apop_opts.db_engine=='m') ? tab_create_mysql : tab_create_sqlite)(tabname, has_row_names=='y', field_params, table_params, fn),\n return -1, 0, \"Creating the table in the database failed.\");\n#if SQLITE_VERSION_NUMBER < 3003009\n Apop_notify(1, \"Apophenia was compiled using a version of SQLite from mid-2007 or earlier. \"\n \"The code for reading in text files using such an old version is no longer supported, \"\n \"so if errors crop up please see about installing a more recent version of SQLite's library.\");\n#endif\n int use_sqlite_prepared_statements = apop_use_sqlite_prepared_statements(col_ct);\n if (use_sqlite_prepared_statements)\n Apop_stopif(apop_prepare_prepared_statements(tabname, col_ct, &statement), \n return -1, 0, \"Trouble preparing the prepared statement for SQLite.\");\n //done with table & query setup.\n //convert a data line into SQL: insert into TAB values (0.3, 7, \"et cetera\");\n\twhile(L.ct && !L.eof){\n line_to_insert(L, add_this_line, tabname, statement, rows);\n if (apop_opts.verbose > 1 && !(ct++ % batch_size)) \n {fprintf(stderr, \".\"); fflush(NULL);}\n if (use_sqlite_prepared_statements){\n int err = sqlite3_step(statement);\n if (err!=0 && err != 101) //0=ok, 101=done\n Apop_notify(0, \"sqlite insert query gave error code %i.\\n\", err);\n Apop_assert_c(!sqlite3_reset(statement), -1, apop_errorlevel, \"SQLite error.\");\n#if SQLITE_VERSION_NUMBER >= 3003009\n Apop_assert_c(!sqlite3_clear_bindings(statement), -1, apop_errorlevel, \"SQLite error.\"); //needed for NULLs\n#endif\n }\n do {\n L = parse_a_line(infile, buffer, &ptr, add_this_line, field_ends, delimiters);\n rows ++;\n } while (!L.ct && !L.eof); //skip blank lines\n\t}\n apop_data_free(add_this_line);\n#if SQLITE_VERSION_NUMBER >= 3003009\n\tif (use_sqlite_prepared_statements){\n Apop_assert_c(sqlite3_finalize(statement) ==SQLITE_OK, -1, apop_errorlevel, \"SQLite error.\");\n }\n#endif\n if (strcmp(text_file,\"-\")) fclose(infile);\n\treturn rows;\n}\n" }, { "path": "apop_data.m4.c", "content": "/** \\file \nThe apop_data structure joins together a gsl_matrix, apop_name, and a table of strings. */\n/* Copyright (c) 2006--2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n//apop_gsl_error is in apop_linear_algebra.c\n#define Set_gsl_handler gsl_error_handler_t *prior_handler = gsl_set_error_handler(apop_gsl_error);\n#define Unset_gsl_handler gsl_set_error_handler(prior_handler);\n\n/** Allocate an \\ref apop_data structure.\n \n\\li The typical case is three arguments, like apop_data_alloc(2,3,4): vector size, matrix rows, matrix cols. If the first argument is zero, you get a \\c NULL vector.\n\\li Two arguments, apop_data_alloc(2,3), would allocate just a matrix, leaving the vector \\c NULL.\n\\li One argument, apop_data_alloc(2), would allocate just a vector, leaving the matrix \\c NULL.\n\\li Zero arguments, apop_data_alloc(), will produce a basically blank set, with \\c out->matrix and \\c out->vector set to \\c NULL. \n\nFor allocating the text part, see \\ref apop_text_alloc.\n\nThe \\c weights vector is set to \\c NULL. If you need it, allocate it via\n\\code d->weights = gsl_vector_alloc(row_ct); \\endcode\n\n\\return The \\ref apop_data structure, allocated and ready to be populated with data.\n\\exception out->error=='a' Allocation error. The matrix, vector, or names couldn't be malloced, which probably means that you requested a very large data set.\n\n\\li An \\ref apop_data struct, by itself, is about 72 bytes. If I can't allocate that much memory, I return \\c NULL.\n But if even this much fails, your computer may be on fire and you should go put it out. \n\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\see apop_data_calloc\n*/\nAPOP_VAR_HEAD apop_data * apop_data_alloc(const size_t size1, const size_t size2, const int size3){\n const size_t apop_varad_var(size1, 0);\n const size_t apop_varad_var(size2, 0);\n const int apop_varad_var(size3, 0);\nAPOP_VAR_ENDHEAD\n size_t vsize=0, msize1=0; \n int msize2=0;\n if (size3){\n vsize = size1;\n msize1 = size2;\n msize2 = size3;\n }\n else if (size2) {\n msize1 = size1;\n msize2 = size2;\n }\n else vsize = size1;\n apop_data *setme = malloc(sizeof(apop_data));\n Apop_stopif(!setme, return NULL, -5, \"malloc failed. Probably out of memory.\");\n *setme = (apop_data) { }; //init to zero/NULL.\n Set_gsl_handler\n if (msize2 > 0 && msize1 > 0){\n setme->matrix = gsl_matrix_alloc(msize1,msize2);\n Apop_stopif(!setme->matrix, setme->error='a'; return setme,\n 0, \"malloc failed on a %zu x %i matrix. Probably out of memory.\", msize1, msize2);\n }\n if (vsize){\n setme->vector = gsl_vector_alloc(vsize);\n Apop_stopif(!setme->vector, setme->error='a'; return setme,\n 0, \"malloc failed on a vector of size %zu. Probably out of memory.\", vsize);\n }\n Unset_gsl_handler\n setme->names = apop_name_alloc();\n Apop_stopif(!setme->names, setme->error='a'; return setme,\n 0, \"couldn't allocate names. Probably out of memory.\");\n return setme;\n}\n\n/** Allocate a \\ref apop_data structure, to be filled with data; set everything in the allocated portion to zero. See \\ref apop_data_alloc for details.\n\n\\return The \\ref apop_data structure, allocated and zeroed out.\n\\exception out->error=='a' allocation error; probably out of memory.\n\\li This function uses the \\ref designated syntax for inputs.\n\\see apop_data_alloc \n*/\nAPOP_VAR_HEAD apop_data * apop_data_calloc(const size_t size1, const size_t size2, const int size3){\n const size_t apop_varad_var(size1, 0);\n const size_t apop_varad_var(size2, 0);\n const int apop_varad_var(size3, 0);\nAPOP_VAR_ENDHEAD\n size_t vsize=0, msize1=0; \n int msize2=0;\n if (size3){\n vsize = size1;\n msize1 = size2;\n msize2 = size3;\n }\n else if (size2) {\n msize1 = size1;\n msize2 = size2;\n }\n else vsize = size1;\n apop_data *setme = malloc(sizeof(apop_data));\n Apop_stopif(!setme, apop_return_data_error('a'), 0, \"malloc failed. Probably out of memory.\");\n *setme = (apop_data) { }; //init to zero/NULL.\n if (msize2 >0 && msize1 > 0){\n setme->matrix = gsl_matrix_calloc(msize1,msize2);\n Apop_stopif(!setme->matrix, apop_return_data_error('a'), 0, \"malloc failed on a %zu x %i matrix. Probably out of memory.\", msize1, msize2);\n }\n if (vsize){\n setme->vector = gsl_vector_calloc(vsize);\n Apop_stopif(!setme->vector, apop_return_data_error('a'), 0, \"malloc failed on a vector of size %zu. Probably out of memory.\", vsize);\n }\n setme->names = apop_name_alloc();\n return setme;\n}\n\n/*For a touch of space saving, blank strings in a text grid \nall point to the same nul string. */\nchar *apop_nul_string = \"\";\n\nstatic void apop_text_blank(apop_data *in, const size_t row, const size_t col){\n if (in->text[row][col] != apop_nul_string) free(in->text[row][col]);\n in->text[row][col] = apop_nul_string;\n}\n\n/** Free a matrix of chars* (i.e., a char***).\nThis is what \\c apop_data_free uses internally to deallocate the \\c text element of\nan \\ref apop_data set. You may never need to use it directly.\n\nSample usage:\n\\code\napop_text_free(yourdata->text, yourdata->textsize[0], yourdata->textsize[1]);\n\\endcode\n*/\nvoid apop_text_free(char ***freeme, int rows, int cols){\n if (rows && cols)\n for (int i=0; i < rows; i++){\n for (int j=0; j < cols; j++)\n if(freeme[i][j]!=apop_nul_string) \n free(freeme[i][j]);\n free(freeme[i]);\n }\n free(freeme);\n}\n\n/** Free the elements of the given \\ref apop_data set and then the \\ref apop_data set\n itself. Intended to be used by \\ref apop_data_free, a macro that calls this to free\n elements, then sets the value to \\c NULL.\n\n\\li \\ref apop_data_free is a macro that calls this function and, on success, sets the input pointer to \\c NULL. \nFor typical cases, that's slightly more useful than this function.\n\n\\exception freeme.error='c' Circular linking is against the rules. If freeme->more == freeme, then \nI set freeme.error='c' and return. If you send in a structure like A -> B ->\nB, then both data sets A and B will be marked.\n\n\\return \\c 0 on OK, \\c 'c' on error.\n*/\nchar apop_data_free_base(apop_data *freeme){\n if (!freeme) return 0;\n if (freeme->more){\n Apop_stopif(freeme == freeme->more, freeme->error='c'; return 'c',\n 1, \"the ->more element of this data set equals the data set itself. \"\n \"This is not healthy. Not freeing; marking your data set with error='c'.\");\n if (apop_data_free_base(freeme->more)) \n Apop_stopif(freeme->more->error == 'c', freeme->error='c'; return 'c', \n 1, \"Propogating error code to parent data set\");\n } \n if (freeme->vector) \n gsl_vector_free(freeme->vector);\n if (freeme->matrix) \n gsl_matrix_free(freeme->matrix); \n if (freeme->weights)\n gsl_vector_free(freeme->weights);\n apop_name_free(freeme->names);\n apop_text_free(freeme->text, freeme->textsize[0] , freeme->textsize[1]);\n free(freeme);\n return 0;\n}\n\n/** Copy one \\ref apop_data structure to another.\n\nThis function does not allocate the output structure or the vector, matrix, text,\nor weights elements---I assume you have already done this and got the dimensions\nright. I will assert that there is at least enough room in the destination for your\ndata, and fail if the copy would write more elements than there are bins.\n\n \\li If you want space allocated or are unsure about dimensions, use \\ref apop_data_copy.\n \\li If both \\c in and \\c out have a \\c more pointer, also copy subsequent page(s).\n \\li You can use the subsetting macros, \\ref Apop_r, \\ref Apop_rs, \\ref Apop_c,\n and so on, to copy within a data set:\n \\li Copying a \\c NULL to a \\c NULL is valid but does nothing. Other attempt to write\n to a \\c NULL fail with an error printed to stdout if apop_opts.verbose >= 1.\n\n\\code\n//Copy the contents of row i of mydata to row j.\napop_data *fromrow = Apop_r(mydata, i);\napop_data *torow = Apop_r(mydata, j);\napop_data_memcpy(torow, fromrow);\n\n// or just\napop_data_memcpy(Apop_r(mydata, i), Apop_r(mydata, j));\n\\endcode\n \n \\param out A structure that this function will fill. Must be preallocated with the appropriate sizes.\n \\param in The input data.\n\n\\exception out.error='d' Dimension error.\n\\exception out.error='p' Part missing; e.g., in->matrix exists but out->matrix doesn't.\n*/\nvoid apop_data_memcpy(apop_data *out, const apop_data *in){\n if (!out && !in) return;\n Apop_stopif(!out, return, 0, \"you are copying to a NULL matrix. Do you mean to use apop_data_copy instead?\");\n Apop_stopif(out==in, return, 1, \"out==in. Doing nothing.\");\n if (in->matrix){\n Apop_stopif(!out->matrix, out->error='p'; return, 1, \"in->matrix exists but out->matrix does not.\");\n Apop_stopif(in->matrix->size1 != out->matrix->size1 || in->matrix->size2 != out->matrix->size2, \n out->error='d'; return,\n 1, \"you're trying to copy a (%zu X %zu) into a (%zu X %zu) matrix.\", \n in->matrix->size1, in->matrix->size2, out->matrix->size1, out->matrix->size2);\n gsl_matrix_memcpy(out->matrix, in->matrix);\n }\n if (in->vector){\n Apop_stopif(!out->vector, out->error='p'; return, 1, \"in->vector exists but out->vector does not.\");\n Apop_stopif(in->vector->size != out->vector->size,\n out->error='d'; return,\n 1, \"You're trying to copy a %zu-elmt \"\n \"vector into a %zu-elmt vector.\", in->vector->size, out->vector->size);\n gsl_vector_memcpy(out->vector, in->vector);\n }\n if (in->weights){\n Apop_stopif(!out->weights, out->error='p'; return, 1, \"in->weights exists but out->weights does not.\");\n Apop_stopif(in->weights->size != out->weights->size,\n out->error='d'; return,\n 1, \"Weight vector sizes don't match: \"\n \"you're trying to copy a %zu-elmt vector into a %zu-elmt vector.\", \n in->weights->size, out->weights->size);\n gsl_vector_memcpy(out->weights, in->weights);\n }\n if (in->names){\n if (!out->names) out->names = apop_name_alloc();\n Asprintf(&out->names->title, \"%s\", in->names->title);\n if (out->names->vector && in->names->vector) {Asprintf(&out->names->vector, \"%s\", in->names->vector);}\n for (int i=0; i< in->names->rowct; i++)\n if (i< out->names->rowct) {Asprintf(out->names->row+i, \"%s\", in->names->row[i]);}\n else apop_name_add(out->names, in->names->row[i], 'r');\n for (int i=0; i< in->names->colct; i++)\n if (i< out->names->colct) {Asprintf(out->names->col+i, \"%s\", in->names->col[i]);}\n else apop_name_add(out->names, in->names->col[i], 'c');\n for (int i=0; i< in->names->textct; i++)\n if (i< out->names->textct) {Asprintf(out->names->text+i, \"%s\", in->names->text[i]);}\n else apop_name_add(out->names, in->names->text[i], 't');\n }\n out->textsize[0] = in->textsize[0]; \n out->textsize[1] = in->textsize[1]; \n if (in->textsize[0] && in->textsize[1]){\n Apop_stopif(out->textsize[0] < in->textsize[0] || out->textsize[1] < in->textsize[1],\n out->error='d'; return,\n 1, \"I am trying to copy a grid of (%zu, %zu) text elements into a grid of (%zu, %zu), \"\n \"and that won't work. Please use apop_text_alloc to reallocate the right amount of data, \"\n \"or use apop_data_copy for automatic allocation.\",\n in->textsize[0] , in->textsize[1] , out->textsize[0] , out->textsize[1]);\n for (size_t i=0; i< in->textsize[0]; i++)\n for(size_t j=0; j < in->textsize[1]; j ++)\n if (in->text[i][j] == apop_nul_string)\n apop_text_blank(out, i, j);\n else apop_text_set(out, i, j, \"%s\", in->text[i][j]);\n }\n if (in->more && out->more) apop_data_memcpy(out->more, in->more);\n}\n\n/** Copy one \\ref apop_data structure to another. That is, all data is duplicated.\n\nBasically a front-end for \\ref apop_data_memcpy for those who prefer this sort of syntax. \n\nIf the data set has a \\c more pointer, that will be followed and subsequent pages copied as well.\n \n \\param in the input data\n \\return a structure that this function will allocate and fill. If input is NULL, then this will be NULL.\n\n\\exception out.error='a' Allocation error.\n\\exception out.error='c' Cyclic link: D->more == D (may be later in the chain, e.g., D->more->more = D->more) You'll have only a partial copy.\n\\exception out.error='d' Dimension error; should never happen.\n\\exception out.error='p' Missing part error; should never happen.\n\n\\li If the input data set has an error, then I will copy it anyway, including the\nerror flag (which might be overwritten). I print a warning if the verbosity level\nis >=1.\n\n */\napop_data *apop_data_copy(const apop_data *in){\n if (!in) return NULL;\n apop_data *out = apop_data_alloc();\n Apop_stopif(out->error, return out, 0, \"Allocation error.\");\n if (in->error){\n Apop_notify(1, \"the data set to be copied has an error flag of %c. Copying it.\", in->error);\n out->error = in->error;\n }\n if (in->more){\n Apop_stopif(in == in->more, out->error='c'; return out,\n 0, \"the ->more element of this data set equals the \"\n \"data set itself. This is not healthy. Made a partial copy and set out.error='c'.\");\n out->more = apop_data_copy(in->more);\n Apop_stopif(out->more->error, out->error=out->more->error; return out,\n 0, \"propagating an error in the ->more element to the parent apop_data set. Only a partial copy made.\");\n }\n if (in->vector){\n out->vector = gsl_vector_alloc(in->vector->size);\n Apop_stopif(!out->vector, out->error='a'; return out, 0, \"Allocation error on vector of size %zu.\", in->vector->size);\n }\n if (in->matrix){ \n out->matrix = gsl_matrix_alloc(in->matrix->size1, in->matrix->size2);\n Apop_stopif(!out->matrix, out->error='a'; return out, 0, \"Allocation error on matrix \"\n \"of size %zu X %zu.\", in->matrix->size1, in->matrix->size2);\n }\n if (in->weights){\n out->weights = gsl_vector_alloc(in->weights->size);\n Apop_stopif(!out->weights, out->error='a'; return out, 0, \"Allocation error on weights vector of size %zu.\", in->weights->size);\n }\n if (in->textsize[0] && in->textsize[1]){\n apop_text_alloc(out, in->textsize[0], in->textsize[1]);\n Apop_stopif(out->error, return out, 0, \"Allocation error on text grid of size %zu X %zu.\", in->textsize[0], in->textsize[1]);\n }\n apop_data_memcpy(out, in);\n return out;\n}\n\n/** Put the first data set either on top of or to the left of the second data set.\n\nFor the opposite operation, see \\ref apop_data_split.\n\n\\param m1 the upper/rightmost data set (default = \\c NULL)\n\\param m2 the second data set (default = \\c NULL)\n\\param posn If 'r', stack rows of m1 above rows of m2
\n if 'c', stack columns of m1 to left of m2's
\n (default = 'r')\n\\param inplace If \\c 'y', use \\ref apop_matrix_realloc and \\ref apop_vector_realloc to modify \\c m1 in place. Otherwise, allocate a new \\ref apop_data set, leaving \\c m1 undisturbed. (default='n')\n\\return The stacked data, either in a new \\ref apop_data set or \\c m1\n\\exception out->error=='a' Allocation error.\n\\exception out->error=='d' Dimension error; couldn't make a complete copy.\n\n\\li The function returns a new data set, meaning that until you apop_data_free()\n the original data sets, you will be taking up twice as much memory.\n\\li If m1 or m2 are \\c NULL, returns a copy of the other element, and if\n both are \\c NULL, returns \\c NULL. If \\c m2 is \\c NULL and \\c inplace is \\c\n 'y', returns the original \\c m1 pointer unmodified.\n\\li Text is handled as you'd expect: If 'r', one set of text is stacked on top of the\n other [number of columns must match]; if 'c', one set of text is set next to the other\n [number of rows must match].\n\\li \\c more is ignored.\n\\li If stacking rows on rows, the output vector is the input\n vectors stacked accordingly. If stacking columns by columns, the output\n vector is just a copy of the vector of \\c m1 and m2->vector doesn't appear in the\n output at all. \n\\li The same rules for dealing with the vector(s) hold for the vector(s) of weights.\n\\li Names are a copy of the names for \\c m1, with the names for \\c m2 appended to the\n row or column list, as appropriate.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_data_stack(apop_data *m1, apop_data * m2, char posn, char inplace){\n apop_data * apop_varad_var(m1, NULL)\n apop_data * apop_varad_var(m2, NULL)\n char apop_varad_var(posn, 'r')\n Apop_stopif(!(posn == 'r' || posn == 'c'), return NULL, 0, \"Valid positions are 'r' or 'c'\"\n \" you gave me '%c'. Returning NULL.\", posn);\n char apop_varad_var(inplace, 'n')\n inplace = (inplace == 'y' || inplace == 1 || inplace == 'Y') ? 1 : 0;\nAPOP_VAR_ENDHEAD\n if (!m1) return apop_data_copy(m2);\n if (!m2) return inplace ? m1 : apop_data_copy(m1);\n apop_data *out = NULL;\n if (inplace)\n out = m1;\n else {\n apop_data *m = m1->more; //not following the more pointer.\n m1->more =NULL;\n out = apop_data_copy(m1);\n Apop_stopif(out->error, return out, 0, \"initial copy failed; leaving.\");\n m1->more = m;\n }\n Get_vmsizes(m1); //original sizes of vsize, msize1, msize2.\n if (m2->names && !out->names) out->names = apop_name_alloc();\n \n if (posn == 'c'){\n if (m2->vector && out->vector){\n gsl_matrix_view mview = gsl_matrix_view_vector(m2->vector, m2->vector->size, 1);\n out->matrix = apop_matrix_stack(out->matrix, &mview.matrix, posn, .inplace='y');\n apop_name_stack(out->names, m2->names, 'c', 'v');\n if (m2->names && !m2->names->vector && m2->names->colct) apop_name_add(out->names, \"v\", 'c');\n }\n if (m2->vector && !out->vector) {\n out->vector= apop_vector_copy(m2->vector);\n if (m2->names->vector) apop_name_add(out->names, m2->names->vector, 'v');\n }\n }\n\n out->matrix = apop_matrix_stack(out->matrix, m2->matrix, posn, .inplace='y');\n\n\n if (posn == 'r'){\n out->vector = apop_vector_stack(out->vector, m2->vector, .inplace='y');\n out->weights = apop_vector_stack(out->weights, m2->weights, .inplace='y');\n } \n\n if (m2->text){ //we've already copied m1->text, if any, so if m2->text is NULL, we're done.\n if (posn=='r'){\n Apop_stopif(out->text && m2->textsize[1]!=out->textsize[1], \n out->error='d'; return out, 0,\n \"The first data set has %zu columns of text and the second has %zu columns. \"\n \"I can't stack that.\", out->textsize[1], m2->textsize[1]);\n int basetextsize = out->textsize[0];\n apop_text_alloc(out, basetextsize+m2->textsize[0], m2->textsize[1]);\n Apop_stopif(out->error, return out, 0, \"Allocation error.\");\n for(int i=0; i< m2->textsize[0]; i++)\n for(int j=0; j< m2->textsize[1]; j++)\n if (m2->text[i][j] == apop_nul_string)\n apop_text_blank(out, i+basetextsize, j);\n else apop_text_set(out, i+basetextsize, j, \"%s\", m2->text[i][j]);\n } else {\n Apop_stopif(out->text && m2->textsize[0]!=out->textsize[0], \n out->error='d'; return out, 0,\n \"The first data set has %zu rows of text and the second has %zu rows. \"\n \"I can't stack that.\", out->textsize[0], m2->textsize[0]);\n int basetextsize = out->textsize[1];\n apop_text_alloc(out, m2->textsize[0], basetextsize+m2->textsize[1]);\n Apop_stopif(out->error, out->error='a'; return out, 0, \"Allocation error.\");\n for(int i=0; i< m2->textsize[0]; i++)\n for(int j=0; j< m2->textsize[1]; j++)\n if (m2->text[i][j] == apop_nul_string)\n apop_text_blank(out, i, j+basetextsize);\n else apop_text_set(out, i, j+basetextsize, \"%s\", m2->text[i][j]);\n apop_name_stack(out->names, m2->names, 't');\n }\n }\n if ((posn=='r' && m2->names && m2->names->rowct) || (posn=='c' && m2->names && m2->names->colct)){\n int min = posn =='r' ? m1->names->rowct : m1->names->colct;\n int max = posn =='r' ? GSL_MAX(vsize, msize1) : msize2;\n for (int k = min; k < max; k++) //pad so the name stacking is aligned (if needed)\n apop_name_add(out->names, \"\", posn); \n apop_name_stack(out->names, m2->names, posn);\n }\n return out;\n}\n\n/** Split one input \\ref apop_data structure into two.\n\n For the opposite operation, see \\ref apop_data_stack.\n \n\\param in The \\ref apop_data structure to split \n\\param splitpoint The index of what will be the first row/column of the second data set.\nE.g., if this is -1 and \\c r_or_c=='c', then the whole data set will be in the second\ndata set; if this is the length of the matrix then the whole data set will be in the\nfirst data set. Another way to put it is that for values between zero and the matrix's\nsize, \\c splitpoint will equal the number of rows/columns in the first matrix.\n\n\\param r_or_c If this is 'r' or 'R', then put some rows in the first data set and some in the second; of 'c' or 'C', split columns into first and second data sets.\n\n \\return An array of two \\ref apop_data sets. If one is empty then a\n \\c NULL pointer will be returned in that position. For example, for a data set of 50 rows, apop_data **out = apop_data_split(data, 100, 'r') sets out[0] = apop_data_copy(data) and out[1] = NULL.\n\n \\li When splitting at a row, the text is also split.\n \\li The \\c more pointer is ignored.\n \\li The apop_data->vector is taken to be the -1st element of the matrix. \n \\li Weights will be preserved. If splitting by rows, then the top and bottom parts of the weights vector will be assigned to the top and bottom parts of the main data set. If splitting by columns, identical copies of the weights vector will be assigned to both parts.\n \\li Data is copied, so you may want to call apop_data_free(in) after this.\n */\napop_data ** apop_data_split(apop_data *in, int splitpoint, char r_or_c){\n //A long, dull series of contingencies. Bonus: a reasonable use of goto.\n apop_data **out = malloc(2*sizeof(apop_data *));\n out[0] = out[1] = NULL;\n Apop_stopif(!in, return out, 1, \"input was NULL; output will be an array of two NULLs.\");\n gsl_vector v1, v2, w1, w2;\n gsl_matrix m1, m2;\n int set_v1 = 1, set_v2 = 1,\n set_m1 = 1, set_m2 = 1,\n set_w1 = 1, set_w2 = 1,\n namev0 = 0, namev1 = 0,\n namer0 = 0, namer1 = 0,\n namec0 = 0, namec1 = 0,\n namersplit = -1, namecsplit = -1;\n if (r_or_c == 'r' || r_or_c == 'R') {\n if (splitpoint <=0)\n out[1] = apop_data_copy(in);\n else if (in->matrix && splitpoint >= in->matrix->size1)\n out[0] = apop_data_copy(in);\n else {\n namev0 =\n namev1 = \n namec0 =\n namec1 = 1;\n if (in->vector){\n v1 = gsl_vector_subvector(in->vector, 0, splitpoint).vector;\n v2 = gsl_vector_subvector(in->vector, splitpoint, in->vector->size - splitpoint).vector;\n } else\n set_v1 = \n set_v2 = 0;\n if (in->weights){\n w1 = gsl_vector_subvector(in->weights, 0, splitpoint).vector;\n w2 = gsl_vector_subvector(in->weights, splitpoint,\n in->weights->size - splitpoint).vector;\n } else\n set_w1 = \n set_w2 = 0;\n if (in->matrix){\n m1 = gsl_matrix_submatrix (in->matrix, 0, 0, splitpoint, in->matrix->size2).matrix;\n m2 = gsl_matrix_submatrix (in->matrix, splitpoint, 0,\n in->matrix->size1 - splitpoint, in->matrix->size2).matrix;\n } else\n set_m1 = \n set_m2 = 0;\n namersplit=splitpoint;\n goto allocation;\n }\n } else if (r_or_c == 'c' || r_or_c == 'C') {\n if (in->weights){\n w1 = gsl_vector_subvector(in->weights, 0, in->weights->size).vector;\n w2 = gsl_vector_subvector(in->weights, 0, in->weights->size).vector;\n } else \n set_w1 = \n set_w2 = 0;\n namer0 = 1;\n namer1 = 1;\n\n if (splitpoint <= -1)\n out[1] = apop_data_copy(in);\n else if (in->matrix && splitpoint >= in->matrix->size2)\n out[0] = apop_data_copy(in);\n else if (splitpoint == 0){\n if (in->vector){\n v1 = gsl_vector_subvector(in->vector, 0, in->vector->size).vector;\n namev0 = 1;\n } else \n set_v1 = 0;\n set_v2 = 0;\n set_m1 = 0;\n if (in->matrix){\n m2 = gsl_matrix_submatrix (in->matrix, 0, 0, \n in->matrix->size1, in->matrix->size2).matrix;\n namec1 = 1;\n } else \n set_m2 = 0;\n goto allocation;\n } else if (splitpoint > 0 && in->matrix && splitpoint < in->matrix->size2){\n if (in->vector){\n v1 = gsl_vector_subvector(in->vector, 0, in->vector->size).vector;\n namev0 = 1;\n } else \n set_v1 = 0;\n set_v2 = 0;\n if (in->matrix){\n m1 = gsl_matrix_submatrix (in->matrix, 0, 0, in->matrix->size1, splitpoint).matrix;\n m2 = gsl_matrix_submatrix (in->matrix, 0, splitpoint, \n in->matrix->size1, in->matrix->size2-splitpoint).matrix;\n namecsplit = splitpoint;\n } else\n set_m1 = \n set_m2 = 0;\n goto allocation;\n } else { //splitpoint >= in->matrix->size2\n if (in->vector){\n v1 = gsl_vector_subvector(in->vector, 0, in->vector->size).vector;\n namev0 = 1;\n } else \n set_v1 = 0;\n set_v2 = 0;\n if (in->matrix){\n m1 = gsl_matrix_submatrix (in->matrix, 0, 0, \n in->matrix->size1, in->matrix->size2).matrix;\n namec0 = 1;\n }\n else set_m1 = 0;\n set_m2 = 0;\n goto allocation;\n }\n } else Apop_notify(0, \"Please set r_or_c == 'r' or == 'c'. Returning two NULLs.\");\n return out;\n\nallocation:\n out[0] = apop_data_alloc();\n out[1] = apop_data_alloc();\n if (set_v1) out[0]->vector = apop_vector_copy(&v1);\n if (set_v2) out[1]->vector = apop_vector_copy(&v2);\n if (set_m1) out[0]->matrix = apop_matrix_copy(&m1);\n if (set_m2) out[1]->matrix = apop_matrix_copy(&m2);\n if (set_w1) out[0]->weights = apop_vector_copy(&w1);\n if (set_w2) out[1]->weights = apop_vector_copy(&w2);\n if (namev0 && out[0]) apop_name_stack(out[0]->names, in->names, 'v');\n if (namev1 && out[1]) apop_name_stack(out[1]->names, in->names, 'v');\n if (namersplit >=0)\n for (int k=0; k< in->names->rowct; k++){\n int which = (k >= namersplit);\n assert(out[which]);\n apop_name_add(out[which]->names, in->names->row[k], 'r');\n }\n else {\n if (namer0 && out[0]) apop_name_stack(out[0]->names, in->names, 'r');\n if (namer1 && out[1]) apop_name_stack(out[1]->names, in->names, 'r');\n }\n if (namecsplit >=0)\n for (int k=0; k< in->names->colct; k++){\n int which = (k >= namecsplit);\n assert(out[which]);\n apop_name_add(out[which]->names, in->names->col[k], 'c');\n }\n else {\n if (namec0 && out[0]) apop_name_stack(out[0]->names, in->names, 'c');\n if (namec1 && out[1]) apop_name_stack(out[1]->names, in->names, 'c');\n }\n //finally, the text [split by rows only]\n if (r_or_c=='r' && in->textsize[0] && in->textsize[1]){\n apop_name_stack(out[1]->names, in->names, 't');\n apop_text_alloc(out[0], splitpoint, in->textsize[1]);\n Apop_stopif(out[0]->error, return out, 0, \"Allocation error.\");\n if (in->textsize[0] > splitpoint){\n apop_name_stack(out[0]->names, in->names, 't');\n apop_text_alloc(out[1], in->textsize[0]-splitpoint, in->textsize[1]);\n Apop_stopif(out[1]->error, return out, 0, \"Allocation error.\");\n }\n for (int i=0; i< in->textsize[0]; i++)\n for (int j=0; j< in->textsize[1]; j++){\n int whichtext = (i >= splitpoint);\n int row = whichtext ? i - splitpoint : i;\n Asprintf(&(out[whichtext]->text[row][j]), \"%s\", in->text[i][j]);\n }\n }\n return out;\n}\n\n\n/** Remove the columns set to one in the \\c drop vector.\n\\param n the \\ref apop_name structure to be pared down\n\\param drop a vector with n->colct elements, mostly zero, with a one marking those columns to be removed.\n\\see \\ref apop_data_prune_columns\n*/\nstatic void apop_name_rm_columns(apop_name *n, int *drop){\n apop_name *newname = apop_name_alloc();\n size_t initial_colct = n->colct;\n for (size_t i=0; i< initial_colct; i++){\n if (drop[i]==0) apop_name_add(newname, n->col[i],'c');\n else n->colct--;\n free(n->col[i]);\n }\n free(n->col);\n n->col = newname->col;\n\n //we need to free the newname struct, but leave the column intact.\n newname->col = NULL;\n newname->colct = 0;\n apop_name_free(newname);\n}\n\n\nstatic gsl_matrix *apop_matrix_rm_columns(gsl_matrix *in, int *drop){\n int ct = 0, //how many columns will not be dropped?\n j = 0;\n for (size_t i=0; i < in->size2; i++)\n if (drop[i]==0)\n ct++;\n if (ct == in->size2) return apop_matrix_copy(in);\n if (ct == 0) return NULL;\n gsl_matrix *out = gsl_matrix_alloc(in->size1, ct);\n for (size_t i=0; i < in->size2; i++){\n if (drop[i]==0){\n gsl_vector *v = Apop_cv(&(apop_data){.matrix=in}, i);\n gsl_matrix_set_col(out, j, v);\n j ++;\n }\n }\n return out;\n}\n\n/** Remove the columns of the \\ref apop_data set corresponding to a nonzero value in the \\c drop vector.\n\n\\li The returned data structure looks like it was modified in place, but the data\nmatrix and the names are duplicated before being pared down, so if your data is taking\nup more than half of your memory, this may not work.\n\n\\param d The \\ref apop_data structure to be pared down. \n\\param drop An array of ints. If use[7]==1, then column seven will be cut from the\noutput. A reminder: calloc(in->size2 , sizeof(int)) will fill your array with zeros on allocation, and \nmemset(use, 1, in->size2 * sizeof(int)) will\nquickly fill an array of ints with nonzero values.\n\\ref apop_data_rm_rows\n*/\nvoid apop_data_rm_columns(apop_data *d, int *drop){\n gsl_matrix *freeme = d->matrix;\n d->matrix = apop_matrix_rm_columns(d->matrix, drop);\n gsl_matrix_free(freeme); \n apop_name_rm_columns(d->names, drop);\n}\n\n/** \\def apop_data_prune_columns(in, ...)\n Keep only the columns of a data set that you name.\n\n\\param in The data set to prune.\n\\param ... A list of names to retain (i.e. the columns that shouldn't be pruned\nout). For example, if you have run \\ref apop_data_summarize, you have columns for several\nstatistics, but may care about only one or two; see the example.\n\nFor example:\n\\include test_pruning.c \n\n\\li I use a case-insensitive search to find your column.\n\\li If your name multiple columns, I'll only give you the first.\n\\li If I can't find a column matching one of your strings, I throw an error to the screen and continue.\n\\li This is a macro calling \\ref apop_data_prune_columns_base. It packages your list of\ncolumns into a list of strings, adds a \\c NULL string at the end, and calls that function.\n\\hideinitializer */\n \n/** Keep only the columns of a data set that you name.\n This is the function called internally by the \\ref apop_data_prune_columns macro. In\n most cases, you'll want to use that macro. An example of the two uses demonstrating the\n difference:\n\n \\code \n apop_data_prune_columns(d, \"mean\", \"median\");\n\n char *list[] = {\"mean\", \"median\", NULL};\n apop_data_prune_columns_base(d, list);\n \\endcode\n\n\\param d The data set to prune.\n\\param colnames A NULL-terminated list of names to retain. \n\\return A pointer to the input data set, now pruned.\n\\see apop_data_rm_columns\n*/\napop_data* apop_data_prune_columns_base(apop_data *d, char **colnames){\n /* In types.h, you'll find an alias that takes the input, wraps it in the cruft that is\n C's compound literal syntax, and appends a final \"\" to the list of strings. Here, I\n find each element of the list, using that \"\" as a stopper, and then call apop_data_rm_columns.*/\n Apop_stopif(!d, return NULL, 1, \"You're asking me to prune a NULL data set; returning.\");\n Apop_stopif(!d->matrix, return d, 1, \"You're asking me to prune a data set with NULL matrix; returning.\");\n int rm_list[d->names->colct];\n int keep_count = 0;\n char **name_step = colnames;\n //to throw errors for typos (and slight efficiency gains), I need an array of whether\n //each input colname has been used.\n while (*name_step++)\n keep_count++;\n int used_field[keep_count];\n memset(used_field, 0, keep_count*sizeof(int));\n\n for (int i=0; i< d->names->colct; i++){\n int keep = 0;\n for (int j=0; jnames->col[i], colnames[j])){\n keep ++;\n used_field[j]++;\n break;\n }\n rm_list[i] = !keep;\n }\n apop_data_rm_columns(d, rm_list);\n for (int j=0; jrowname==NULL, default is zero.\n\\param col The column number of the desired element. -1 indicates the vector. If colname==NULL, default is zero.\n\\param rowname The row name of the desired element. If NULL, use the row number.\n\\param colname The column name of the desired element. If NULL, use the column number.\n\\param page The case-insensitive name of the page on which the element is found. If \\c NULL, use first page.\n\n\\return A pointer to the element.\n*/\nAPOP_VAR_HEAD double * apop_data_ptr(apop_data *data, int row, int col, const char *rowname, const char *colname, const char *page){\n apop_data * apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, 0, \"You sent me a NULL data set. Returning NULL pointer.\");\n int apop_varad_var(row, 0);\n int apop_varad_var(col, 0);\n const char * apop_varad_var(rowname, NULL);\n const char * apop_varad_var(colname, NULL);\n const char * apop_varad_var(page, NULL);\n\n if (page){\n data = apop_data_get_page(data, page);\n Apop_stopif(!data, return NULL, 1, \"I couldn't find a page with label '%s'. Returning NULL.\", page);\n };\n if (rowname){\n row = apop_name_find(data->names, rowname, 'r');\n Apop_stopif(row == -2, return NULL, 1, \"Couldn't find '%s' amongst the row names.\", rowname);\n }\n if (colname){\n col = apop_name_find(data->names, colname, 'c');\n Apop_stopif(col == -2, return NULL, 1, \"Couldn't find '%s' amongst the column names.\", colname);\n }\nAPOP_VAR_ENDHEAD\n if (col == -1 || (col == 0 && !data->matrix && data->vector)){\n Apop_stopif(!data->vector, return NULL, 1, \"You asked for the vector element (col=-1) but it is NULL. Returning NULL.\");\n return gsl_vector_ptr(data->vector, row);\n } else {\n Apop_stopif(!data->matrix, return NULL, 1, \"You asked for the matrix element (%i, %i) but the matrix is NULL Returning NULL..\", row, col);\n return gsl_matrix_ptr(data->matrix, row,col);\n }\n return NULL;//the main function is blank.\n}\n\n/** Returns the data element at the given point.\n \nIn case of error (probably that you asked for a data point out of bounds), returns \\c NAN.\n See \\ref data_set_get \"the set/get page\" for details and examples.\n\n\\param data The data set. Must not be \\c NULL.\n\\param row The row number of the desired element. If rowname==NULL, default is zero.\n\\param col The column number of the desired element. -1 indicates the vector. \nIf colname==NULL, default is zero if the ->matrix element is not \\c\nNULL and -1 if the ->matrix element is \\c NULL and the ->vector element is not.\n\n\\param rowname The row name of the desired element. If NULL, use the row number.\n\\param colname The column name of the desired element. If NULL, use the column number.\n\\param page The case-insensitive name of the page on which the element is found. If \\c NULL, use first page.\n\n\\return The value at the given location. */\nAPOP_VAR_HEAD double apop_data_get(const apop_data *data, size_t row, int col, const char *rowname, const char *colname, const char *page){\n const apop_data * apop_varad_var(data, NULL);\n Apop_stopif(!data, return NAN, 0, \"You sent me a NULL data set. Returning NaN.\");\n size_t apop_varad_var(row, 0);\n int apop_varad_var(col, 0);\n const char * apop_varad_var(rowname, NULL);\n const char * apop_varad_var(colname, NULL);\n const char * apop_varad_var(page, NULL);\n \n if (page){\n data = apop_data_get_page(data, page);\n Apop_stopif(!data, return NAN, 1, \"I couldn't find a page with label '%s'. Returning NaN.\", page);\n };\n if (rowname){\n row = apop_name_find(data->names, rowname, 'r');\n Apop_stopif(row == -2, return NAN, 1, \"Couldn't find '%s' amongst the row names. Returning NaN.\", rowname);\n }\n if (colname){\n col = apop_name_find(data->names, colname, 'c');\n Apop_stopif(col == -2, return NAN, 1, \"Couldn't find '%s' amongst the column names. Returning NaN.\", colname);\n }\nAPOP_VAR_ENDHEAD\n if (col==-1 || (col == 0 && !data->matrix && data->vector)){\n Apop_stopif(!data->vector, return NAN, 1, \"You asked for the vector element (col=-1) but it is NULL.\");\n return gsl_vector_get(data->vector, row);\n } else {\n Apop_stopif(!data->matrix, return NAN, 1, \"You asked for the matrix element (%zu, %i) but the matrix is NULL.\", row, col);\n return gsl_matrix_get(data->matrix, row, col);\n }\n}\n\n/* The only hint the GSL gives that something failed is that the error-handler is called.\n The error handling function won't let you set an output to the function. So all we\n can do is use a global variable.\n*/\n\nstatic threadlocal int error_for_set; //see apop_internal.h\n\nvoid apop_gsl_error_for_set(const char *reason, const char *file, int line, int gsl_errno){\n Apop_notify(1, \"%s: %s\", file, reason);\n Apop_maybe_abort(1);\n error_for_set = -1;\n}\n\n/** Set a data element.\nSee \\ref data_set_get \"the set/get page\" for details and examples. \n \n \\return 0=OK, -1=error: couldn't find row/column name, or you asked for a location outside the vector/matrix bounds.\n\n\\li The error codes for out-of-bounds errors are thread-safe iff you are have a\nC11-compliant compiler (thanks to the \\c _Thread_local keyword) or a version of GCC with the \\c __thread\nextension enabled.\n\n\\li Set weights via gsl_vector_set(your_data->weights, row, val);.\n\\li Set text elements via \\ref apop_text_set.\n\n\n\\param data The data set. Must not be \\c NULL.\n\\param row The row number of the desired element. If rowname==NULL, default is zero.\n\\param col The column number of the desired element. -1 indicates the vector. If colname==NULL, default is zero.\n\\param rowname The row name of the desired element. If NULL, use the row number.\n\\param colname The column name of the desired element. If NULL, use the column number.\n\\param page The case-insensitive name of the page on which the element is found. If \\c NULL, use first page.\n\\param val The value to give the point.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_data_set(apop_data *data, size_t row, int col, const double val, const char *colname, const char *rowname, const char *page){\n apop_data * apop_varad_var(data, NULL);\n Apop_stopif(!data, return -1, 0, \"You sent me a NULL data set.\");\n size_t apop_varad_var(row, 0);\n int apop_varad_var(col, 0);\n const double apop_varad_var(val, 0);\n const char * apop_varad_var(rowname, NULL);\n const char * apop_varad_var(colname, NULL);\n const char * apop_varad_var(page, NULL);\n \n if (page){\n data = apop_data_get_page((apop_data*)data, page);\n Apop_stopif(!data, return -1, 1, \"I couldn't find a page with label '%s'. Making no changes.\", page);\n }\n if (rowname){\n row = apop_name_find(data->names, rowname, 'r');\n Apop_stopif(row == -2, return -1, 1, \"Couldn't find '%s' amongst the column names. Making no changes.\", rowname);\n }\n if (colname){\n col = apop_name_find(data->names, colname, 'c');\n Apop_stopif(col == -2, return -1, 1, \"Couldn't find '%s' amongst the column names. Making no changes.\", colname);\n }\nAPOP_VAR_ENDHEAD\n Set_gsl_handler\n if (col==-1 || (col == 0 && !data->matrix && data->vector)){\n Apop_stopif(!data->vector, return -1, 1, \"You're trying to set a vector element (row=-1) but the vector is NULL.\");\n gsl_vector_set(data->vector, row, val);\n } else {\n Apop_stopif(!data->matrix, return -1, 1, \"You're trying to set the matrix element (%zu, %i) but the matrix is NULL.\", row, col);\n gsl_matrix_set(data->matrix, row, col, val);\n }\n Unset_gsl_handler\n return error_for_set;\n}\n\n/** A convenience function to add a named element to a data set. Many of Apophenia's\ntesting procedures use this to easily produce a column of named parameters. It is public\nas a convenience.\n\n\\param d The \\ref apop_data structure. Must not be \\c NULL, but may be blank (as per\nallocation via \\ref apop_data_alloc ( ) ).\n\\param name The name to add\n\\param val the value to add to the set.\n\n\\li I use the position of the last non-empty row name to know where to put the value. If\nthere are two names in the data set, then I will put the new name in\nthe third name slot and the data in the third slot in the vector. If\nyou use this function from start to finish in building your list, then you'll be fine.\n\\li If the vector is too short (or \\c NULL), I will call \\ref apop_vector_realloc internally to make space.\n\\li This fits well with the defaults for \\ref apop_data_get. An example:\n\n\\code\napop_data *list = apop_data_alloc();\napop_data_add_named_elmt(list, \"height\", 165);\napop_data_add_named_elmt(list, \"weight\", 60);\n\ndouble height = apop_data_get(list, .rowname=\"height\");\n\n//or\n#define Lookup(dataset, key) apop_data_get(dataset, .rowname=#key)\nheight = Lookup(list, height);\n\\endcode\n*/\nvoid apop_data_add_named_elmt(apop_data *d, char *name, double val){\n Apop_stopif(!d, return, 0, \"You sent me a NULL apop_data set. \"\n \"Maybe allocate with apop_data_alloc() to start.\");\n apop_name_add(d->names, name, 'r');\n if (!d->vector) d->vector = gsl_vector_alloc(1);\n if (d->vector->size < d->names->rowct)\n apop_vector_realloc(d->vector, d->names->rowct);\n gsl_vector_set(d->vector, d->names->rowct-1, val);\n}\n\n//See apop_data_add_names in types.h.\nvoid apop_data_add_names_base(apop_data *d, const char type, char const ** names){\n if (!d->names) d->names = apop_name_alloc();\n for(char const** name = names; *name !=NULL; name++)\n apop_name_add(d->names, *name, type);\n}\n\n\n/** Add a string to the text element of an \\ref apop_data set. If you\n send me a \\c NULL string, I will write the value of apop_opts.nan_string in the given slot.\n If there is already something in that slot, that string is freed, preventing memory leaks.\n\n\\param in The \\ref apop_data set, that already has an allocated \\c text element.\n\\param row The row\n\\param col The column\n\\param fmt The text to write.\n\\param ... You can use a printf-style fmt and follow it with the usual variables to fill in.\n\n\\return 0=OK, -1=error (probably out-of-bounds)\n\n \\li UTF-8 or ASCII text is correctly handled.\n \\li Apophenia follows a general rule of not reallocating behind your back: if\nyour text matrix is currently of size (3,3) and you try to put an item in slot (4,4),\nthen I display an error rather than reallocating the text matrix.\n \\li The string added is a copy (via asprintf), not a pointer to the input(s).\n \\li If there had been a string at the grid point you are writing to,\nthe old one is freed to prevent leaks. Remember this if you had other pointers aliasing\nthat string.\n \\li If an element is \\c NULL, write apop_opts.nan_string at that point. You\nmay prefer to use \"\" to express a blank.\n \\li \\ref apop_text_alloc will reallocate to a new size if you need. For example,\nthis code will fill the diagonals of the text array with a message, resizing as it goes:\n\n\\code\napop_data *list = (something already allocated.);\nfor (int n=0; n < 10; n++){\n apop_text_alloc(list, n+1, n+1);\n apop_text_set(list, n, n, \"This is cell (%i, %i)\", n, n);\n}\n\\endcode\n*/\nint apop_text_set(apop_data *in, const size_t row, const size_t col, const char *fmt, ...){\n Apop_stopif(!in, return -1, 0, \"You asked me to write text to a NULL data set.\");\n Apop_stopif((in->textsize[0] < (int)row+1) || (in->textsize[1] < (int)col+1), return -1, 0, \"You asked me to put the text \"\n \" '%s' at position (%zu, %zu), but the text array has size (%zu, %zu)\\n\", \n fmt, row, col, in->textsize[0], in->textsize[1]);\n if (in->text[row][col] != apop_nul_string) free(in->text[row][col]);\n if (!fmt){\n Asprintf(&(in->text[row][col]), \"%s\", apop_opts.nan_string);\n return 0;\n }\n va_list argp;\n\tva_start(argp, fmt);\n Apop_stopif(vasprintf(&(in->text[row][col]), fmt, argp)==-1, , 0, \"Trouble writing to a string.\");\n\tva_end(argp);\n return 0;\n}\n\n/** This allocates or resizes the \\c text element of an \\ref apop_data set. \n\n If the \\c text element already exists, then this is effectively a \\c realloc function,\n reshaping to the size you specify.\n\n \\param in An \\ref apop_data set. It's OK to send in \\c NULL, in which case an apop_data set with \\c NULL \\c matrix and \\c vector elements is returned.\n \\param row the number of rows of text.\n \\param col the number of columns of text.\n \\return A pointer to the relevant \\ref apop_data set. If the input was not \\c NULL, then this is a repeat of the input pointer.\n \\exception out->error=='a' Allocation error.\n */\napop_data * apop_text_alloc(apop_data *in, const size_t row, const size_t col){\n Apop_stopif((!row && col) || (!col && row), return in, 1, \"Not allocating a %zu x %zu text grid. \"\n \"Returning the input apop_data set.\", row, col);\n if (!in) in = apop_data_alloc();\n if (!in->text){\n if (row){\n in->text = malloc(sizeof(char**) * row);\n Apop_stopif(!in->text, in->error='a'; return in, \n 0, \"malloc failed setting up %zu rows. Probably out of memory.\", row);\n }\n if (row && col)\n for (size_t i=0; i< row; i++){\n in->text[i] = malloc(sizeof(char*) * col);\n Apop_stopif(!in->text[i], in->error='a'; return in, \n 0, \"malloc failed setting up row %zu (with %zu columns). Probably out of memory.\", i, col);\n for (size_t j=0; j< col; j++)\n in->text[i][j] = apop_nul_string;\n }\n } else { //realloc\n size_t rows_now = in->textsize[0];\n size_t cols_now = in->textsize[1];\n if (rows_now > row){\n for (int i=row; i < rows_now; i++){\n for (int j=0; j < cols_now; j++)\n if (in->text[i][j] != apop_nul_string) \n free(in->text[i][j]);\n free(in->text[i]);\n }\n in->text = realloc(in->text, sizeof(char**)*row);\n Apop_stopif(row && !in->text, in->error='a'; return in,\n 0, \"realloc failed shrinking down to %zu rows from %zu rows. \"\n \"There may be actual bugs eating your computer.\", row, rows_now);\n }\n if (rows_now < row){\n in->text = realloc(in->text, sizeof(char**)*row);\n Apop_stopif(!in->text, in->error='a'; return in,\n 0, \"realloc failed setting up %zu rows. Probably out of memory.\", row);\n for (size_t i=rows_now; i < row; i++){\n in->text[i] = malloc(sizeof(char*) * col);\n Apop_stopif(!in->text[i], in->error='a'; return in, \n 0, \"malloc failed setting up row %zu (with %zu columns). Probably out of memory.\", i, col);\n for (int j=0; j < cols_now; j++)\n in->text[i][j] = apop_nul_string;\n }\n }\n if (cols_now > col)\n for (int i=0; i < row; i++)\n for (int j=col; j < cols_now; j++)\n if (in->text[i][j]!=apop_nul_string) \n free(in->text[i][j]);\n if (cols_now != col)\n for (int i=0; i < row; i++){\n in->text[i] = realloc(in->text[i], sizeof(char*)*col);\n for (int j=cols_now; j < col; j++) //happens iff cols_now < col\n in->text[i][j] = apop_nul_string;\n }\n }\n in->textsize[0] = row;\n in->textsize[1] = col;\n return in;\n}\n\n/** Transpose the matrix and text elements of the input data set, including the row/column names. \n\nThe vector and weights elements of the input data set are completely ignored (but see\nalso \\ref apop_vector_to_matrix, which can convert a vector to a 1 X N matrix.) If\ncopying, these other elements won't be present; if .inplace='y', it is up to you to\nhandle these not-transposed elements correctly.\n\n\\param in The input \\ref apop_data set. If \\c NULL, I return \\c NULL. (default: \\c NULL)\n\\param transpose_text If \\c 'y', then also transpose the text element. (default: \\c 'y')\n\\param inplace If \\c 'y', transpose the input in place; if \\c 'n', produce a transposed\ncopy, leaving the original untouched. Due to how gsl_matrix_transpose_memcpy\nworks, a copy will still be made, then copied to the original location. (default: \\c 'y')\n\n\\return If inplace=='n', a newly alloced \\ref apop_data set, with the\nappropriately transposed matrix and/or text. The vector and weights elements will be\n\\c NULL. If transpose_text='n', then the text element of the output set will\nalso be \\c NULL.
if inplace=='y', a pointer to the original data set,\nwith matrix and (if transpose_text='y', text) transposed and vector and weights\nleft in place untouched.\n\n\\li Row names are written to column names of the output matrix, text, or both (whichever is not empty in the input).\n\\li If only the matrix or only the text have names, then the one set of names is written to the row names of the output.\n\\li If both matrix column names and text column names are present, text column names are lost.\n\\li if you have a \\c gsl_matrix with no names or text, you may prefer to use \\c gsl_matrix_transpose_memcpy.\n\\li This function uses the \\ref designated syntax for inputs.\n*/ \nAPOP_VAR_HEAD apop_data * apop_data_transpose(apop_data *in, char transpose_text, char inplace){\n apop_data * apop_varad_var(in, NULL);\n Apop_stopif(!in, return NULL, 1, \"Transposing a NULL data set; returning NULL.\");\n char apop_varad_var(transpose_text, 'y');\n char apop_varad_var(inplace, 'y');\nAPOP_VAR_ENDHEAD\n Apop_stopif(!in->matrix && !*in->textsize, return apop_data_alloc(), \n 1, \"input data set has neither matrix nor text elements; returning an empty data set.\");\n apop_data *out = (inplace=='y') ? in\n : apop_data_alloc(0, in->matrix ? in->matrix->size2 : 0\n , in->matrix ? in->matrix->size1 : 0);\n if (inplace=='y'){\n if (in->matrix) {\n if (in->matrix->size1 == in->matrix->size2)\n gsl_matrix_transpose(in->matrix);\n else {\n gsl_matrix *outm = gsl_matrix_alloc(in->matrix->size2, in->matrix->size1);\n gsl_matrix_transpose_memcpy(outm, in->matrix);\n gsl_matrix_free(in->matrix);\n in->matrix = outm;\n }\n }\n if (out->names){\n char **tmp = out->names->col;\n out->names->col = out->names->row;\n out->names->row = tmp;\n int tmpct = out->names->colct;\n out->names->colct = out->names->rowct;\n out->names->rowct = tmpct;\n }\n } else if (inplace!='y' && in->matrix){\n if (in->matrix) gsl_matrix_transpose_memcpy(out->matrix, in->matrix);\n apop_name_stack(out->names, in->names, 'r', 'c');\n apop_name_stack(out->names, in->names, 'c', 'r');\n }\n if (transpose_text!='y' || in->textsize[0] == 0 || in->textsize[1] == 0) return out;\n if (inplace=='y'){\n size_t orows = in->textsize[0];\n size_t ocols = in->textsize[1];\n if (orows > ocols){ //extend the first ocols rows to their now-longer length\n for (size_t i=0; i< ocols; i++){\n in->text[i] = realloc(in->text[i], sizeof(char*)*orows);\n Apop_stopif(!in->text[i], in->error='a'; return in, \n 0, \"malloc failed setting up row %zu (with %zu columns). Probably out of memory.\", i, orows);\n for (int j=ocols; j < orows; j++)\n in->text[i][j] = in->text[j][i] == apop_nul_string\n ? apop_nul_string\n : strdup(in->text[j][i]);\n }\n }\n if (ocols > orows){ //add rows.\n in->text = realloc(in->text, sizeof(char**)*ocols);\n Apop_stopif(!in->text, in->error='a'; return in,\n 0, \"realloc failed setting up %zu rows. Probably out of memory.\", ocols);\n for (size_t i=orows; i < ocols; i++){\n in->text[i] = malloc(sizeof(char*) * orows);\n Apop_stopif(!in->text[i], in->error='a'; return in, \n 0, \"malloc failed setting up row %zu (with %zu columns). Probably out of memory.\", i, orows);\n for (int j=0; j < orows; j++)\n in->text[i][j] = in->text[j][i] == apop_nul_string\n ? apop_nul_string\n : strdup(in->text[j][i]);\n }\n }\n size_t squaresize = GSL_MIN(orows, ocols);\n for (int i=0; i< squaresize; i++) //now do the no-need-to-extend square\n for (int j=i+1; j< squaresize; j++){\n char *tmp = in->text[i][j];\n in->text[i][j] = in->text[j][i];\n in->text[j][i] = tmp;\n }\n in->textsize[0] = ocols;\n in->textsize[1] = orows;\n } else {\n apop_text_alloc(out, in->textsize[1], in->textsize[0]);\n for (int r=0; r< in->textsize[0]; r++)\n for (int c=0; c< in->textsize[1]; c++)\n if (in->text[r][c] == apop_nul_string)\n apop_text_blank(out, c, r);\n else apop_text_set(out, c, r, in->text[r][c]);\n }\n if (in->names && in->names->textct && !in->names->colct)\n apop_name_stack(out->names, in->names, 't', 'r');\n return out;\n}\n\n/** This function will resize a \\c gsl_matrix to a new height or width.\n\nData in the matrix will be retained. If the new height or width is smaller than the old, then data in the later rows/columns will be cropped away (in a non--memory-leaking manner). If the new height or width is larger than the old, then new cells will be filled with garbage; it is your responsibility to zero out or otherwise fill new rows/columns before use.\n\n \\li A large number of reallocs can take a noticeable amount of time. You\nare encouraged to determine the size of your data beforehand and avoid writing \\c for\nloops that reallocate the matrix at every iteration.\n \\li The gsl_matrix is a versatile struct that can represent submatrices and\nother cuts from parent data. Resizing a subset of a parent matrix makes no sense,\nso return \\c NULL and print a warning if asked to resize a view of a matrix.\n\n\\param m The already-allocated matrix to resize. If you give me \\c NULL, this becomes equivalent to \\c gsl_matrix_alloc\n\\param newheight, newwidth The height and width you'd like the matrix to be.\n\\return m, now resized\n */\ngsl_matrix * apop_matrix_realloc(gsl_matrix *m, size_t newheight, size_t newwidth){\n if (!m)\n return (newheight && newwidth) ? gsl_matrix_alloc(newheight, newwidth) : NULL;\n size_t i, oldoffset=0, newoffset=0, realloced = 0;\n Apop_stopif(m->block->data!=m->data || !m->owner || m->tda != m->size2,\n return NULL, 0, \"I can't resize submatrices or other subviews.\");\n m->block->size = newheight * newwidth;\n if (m->size2 > newwidth)\n for (i=1; i< GSL_MIN(m->size1, newheight); i++){\n oldoffset +=m->size2;\n newoffset +=newwidth;\n memmove(m->data+newoffset, m->data+oldoffset, sizeof(double)*newwidth);\n } \n else if (m->size2 < newwidth){\n m->block->data = m->data = realloc(m->data, sizeof(double) * m->block->size);\n realloced = 1;\n int height = GSL_MIN(m->size1, newheight);\n for (i= height-1; i > 0; i--){\n newoffset +=newwidth;\n memmove(m->data+(height * newwidth) - newoffset, m->data+i*m->size2, sizeof(double)*m->size2);\n }\n }\n m->size1 = newheight;\n m->tda =\n m->size2 = newwidth;\n if (!realloced)\n m->block->data = m->data = realloc(m->data, sizeof(double) * m->block->size);\n return m;\n}\n\n/** This function will resize a \\c gsl_vector to a new length.\n\nData in the vector will be retained. If the new height is\nsmaller than the old, then data at the end of the vector will be\ncropped away (in a non--memory-leaking manner). If the new height is larger than the old,\nthen new cells will be filled with garbage; it is your responsibility\nto zero out or otherwise fill them before use.\n\n \\li A large number of reallocs can take a noticeable amount of time. You\nare thus encouraged to make an effort to determine the size of your data and do one\nallocation, rather than writing \\c for loops that resize a vector at every increment.\n \\li The gsl_vector is a versatile struct that\ncan represent subvectors, matrix columns and other cuts from parent data. \nResizing a portion of a parent matrix makes no sense, so\nreturn \\c NULL and print an error if asked to resize a view.\n\n\\param v The already-allocated vector to resize. If you give me \\c NULL, this is equivalent to \\c gsl_vector_alloc\n\\param newheight The height you'd like the vector to be.\n\\return v, now resized\n */\ngsl_vector * apop_vector_realloc(gsl_vector *v, size_t newheight){\n if (!v) return newheight ? gsl_vector_alloc(newheight) : NULL;\n Apop_stopif(v->block->data!=v->data || !v->owner || v->stride != 1,\n return NULL, 0, \"I can't resize subvectors or other views.\");\n v->block->size = newheight;\n v->size = newheight;\n v->block->data = \n v->data = realloc(v->data, sizeof(double) * v->block->size);\n return v;\n}\n\n/** It's good form to get a page from your data set by name, because you\n may not know the order for the pages, and the stepping through makes\n for dull code anyway (apop_data *page = dataset; while (page->more) page= page->more;).\n\n \\param data The \\ref apop_data set to use. No default; if \\c NULL,\n gives a warning if apop_opts.verbose >=1 and returns \\c NULL.\n\n \\param title The name of the page to retrieve. Default=\\c \"\", which\n is the name of the page of additional estimation information returned\n by estimation routines (log likelihood, status, AIC, BIC, confidence intervals, ...).\n \n \\param match If \\c 'c', case-insensitive match (via \\c strcasecmp); if \\c 'e', exact match, if \\c 'r' regular expression substring search (via \\ref apop_regex). Default=\\c 'c'.\n\n \\return The page whose title matches what you gave me. If I don't find a match, return \\c NULL.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data * apop_data_get_page(const apop_data * data, const char *title, const char match){\n const apop_data * apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, 1, \"You requested a page from a NULL data set. Returning NULL\");\n const char * apop_varad_var(title, \"\");\n const char apop_varad_var(match, 'c');\n Apop_stopif(match!='r' && match!='e' && match!='c', return NULL, 0,\n \"match type needs to be 'r', 'e', or 'c'; you supplied %c.\", match);\nAPOP_VAR_ENDHEAD\n while (data && (!data->names || !data->names->title ||\n (match=='c' && strcasecmp(data->names->title, title))\n || (match=='r' && !apop_regex(data->names->title, title))\n || (match=='e' && strcmp(data->names->title, title))\n ))\n data = data->more;\n return (apop_data *) data; //de-const.\n}\n\n/** Add a page to an \\ref apop_data set. It gets a name so you can find it later.\n\n \\param dataset The input data set, to which a page will be added.\n \\param newpage The page to append\n \\param title The name of the new page.\n\n \\return The new page. I post a warning if I am appending or appending to a \\c NULL data set and apop_opts.verbose >=1 .\n\n \\li See \\ref pps for further notes.\n*/\napop_data * apop_data_add_page(apop_data * dataset, apop_data *newpage, const char *title){\n Apop_stopif(!newpage, return NULL, 1, \"You are adding a NULL page to a data set. Doing nothing; returning NULL.\");\n if (!newpage->names) newpage->names = apop_name_alloc();\n if (title && !(newpage->names->title == title)){//has title, but is not pointing to existing title\n free(newpage->names->title);\n Asprintf(&newpage->names->title, \"%s\", title);\n }\n Apop_stopif(!dataset, return newpage, 1, \"You are adding a page to a NULL data set. Returning the new page as its own data set.\");\n while (dataset->more)\n dataset = dataset->more;\n dataset->more = newpage;\n return newpage;\n}\n\n/** Remove the first page from an \\ref apop_data set that matches a given name.\n\n\\param data The input data set, from which a page will be removed. No default. \nIf \\c NULL, maybe print a warning (see below).\n\n\\param title The case-insensitive name of the page to remove. Default: \\c \"\"\n\\param free_p If \\c 'y', then \\ref apop_data_free the page. Default: \\c 'y'.\n\n\\return If not freed, a pointer to the \\c apop_data page that I just pulled out. Thus,\n you can use this to pull a single page from a data set. I set that page's \\c more\n pointer to \\c NULL, to minimize any confusion about more-than-linear linked list\n topologies. If free_p=='y' (the default) or the page is not found, return \\c NULL.\n\n \\li I don't check the first page, so there's no concern that the head of your list of\n pages will move. Again, the intent of the ->more pointer in the \\ref apop_data\n set is not to fully implement a linked list, but primarily to allow you to staple auxiliary\n information to a main data set.\n\n \\li If I don't find the page you want, I return NULL, and maybe print a warning; see below.\n\n \\li For the two above cases where a warning may be printed, if the page is to be\n returned and apop_opts.verbose >= 1 , print a warning.\n If the page is to be freed and apop_opts.verbose >= 2 , print a warning.\n\n \\li The remaining \\c more pointers in the \\ref apop_data set are adjusted accordingly.\n*/\nAPOP_VAR_HEAD apop_data* apop_data_rm_page(apop_data * data, const char *title, const char free_p){\n const char *apop_varad_var(title, \"\");\n const char apop_varad_var(free_p, 'y');\n apop_data *apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, (free_p=='y'? 2: 1), \"You are removing a \"\n \"page from a NULL a data set. Doing nothing.\");\nAPOP_VAR_ENDHEAD\n while (data->more && strcasecmp(data->more->names->title, title))\n data = data->more;\n Apop_stopif(!data->more, return NULL, (free_p=='y'?2:1), \"You asked me to \"\n \"remove '%s' but I couldn't find a page matching that.\", title);\n if (data->more){\n apop_data *tmp = data->more;\n data->more = data->more->more;\n tmp->more = NULL;\n if (free_p=='y'){\n free(tmp);\n return NULL;\n } //else:\n return tmp;\n } else return NULL;\n}\n\ntypedef int (*apop_fn_ir)(apop_data*, void*);\n\n/** Remove the rows set to one in the \\c drop vector or for which the \\c do_drop function returns one. \n\\param in the \\ref apop_data structure to be pared down\n\\param drop a vector with as many elements as the max of the vector, matrix, or text\n parts of \\c in, with a one marking those rows to be removed.\n\\param do_drop A function that returns one for rows to drop and zero for rows to not drop. A sample function:\n \\code\n int your_drop_function(apop_data *onerow, void *extra_param){\n return gsl_isnan(apop_data_get(onerow)) ||\n !strcmp(onerow->text[0][0], \"Uninteresting data point\");\n }\n \\endcode\n \\ref apop_data_rm_rows will use \\ref Apop_r to get a subview of the input data set\n of height one, and send that subview to this function (and since arguments typically\n default to zero, you don't have to write out things like \\ref apop_data_get\n (onerow, .row=0, .col=0), which can help to keep things readable).\n\\param drop_parameter If your \\c do_drop function requires additional input, put it here\n and it will be passed through.\n\n\\return Returns a pointer to the input data set, now pruned.\n\n\\li If all the rows are to be removed, then you will wind up with the same \\ref\n apop_data set, with \\c NULL \\c vector, \\c matrix, \\c weight, and text. Therefore,\n you may wish to check for \\c NULL elements after use. I remove rownames, but leave\n the other names, in case you want to add new data rows.\n\\li The typical use is to provide only a list or only a function. If both are \\c\n NULL, I return without doing anything, and print a warning if apop_opts.verbose\n >=2. If you provide both, I will drop the row if either the vector has a one in\n that row's position, or if the function returns a nonzero value.\n\\li This function uses the \\ref designated syntax for inputs.\n\\see \\ref apop_data_listwise_delete, \\ref apop_data_rm_columns\n*/ \nAPOP_VAR_HEAD apop_data* apop_data_rm_rows(apop_data *in, int *drop, apop_fn_ir do_drop, void *drop_parameter ){\n apop_data* apop_varad_var(in, NULL);\n Apop_stopif(!in, return in, 2, \"Input data set was NULL; no changes made.\");\n int* apop_varad_var(drop, NULL);\n apop_fn_ir apop_varad_var(do_drop, NULL);\n void* apop_varad_var(drop_parameter, NULL);\n Apop_stopif(!drop && !do_drop, return in, 0, \"You gave me neither a list of ints \"\n \"indicating which rows to drop, nor a drop_fn I can use to test \"\n \"each row. Returning with no changes made.\");\nAPOP_VAR_ENDHEAD\n //First, shift columns down to the nearest not-freed row.\n int outlength = 0;\n Get_vmsizes(in); //vsize, msize1, maxsize\n for (int i=0 ; i < maxsize; i++){\n int drop_row=0;\n if (drop && drop[i]) drop_row = 1;\n else if (do_drop){\n drop_row = do_drop(Apop_r(in, i), drop_parameter);\n }\n if (!drop_row){\n if (outlength != i) apop_data_memcpy(Apop_r(in, outlength), Apop_r(in, i));\n outlength++;\n }\n }\n if (!outlength){\n gsl_vector_free(in->vector); in->vector = NULL;\n gsl_vector_free(in->weights); in->weights = NULL;\n gsl_matrix_free(in->matrix); in->matrix = NULL;\n apop_text_alloc(in, 0, 0);\n //leave colnames intact, remove rownames below.\n }\n\n //now trim excess memory:\n if (in->vector) apop_vector_realloc(in->vector, GSL_MIN(in->vector->size, outlength));\n if (in->weights) apop_vector_realloc(in->weights, GSL_MIN(in->weights->size, outlength));\n if (in->matrix) apop_matrix_realloc(in->matrix, GSL_MIN(in->matrix->size1, outlength), in->matrix->size2);\n if (in->text) apop_text_alloc(in, GSL_MIN(outlength, in->textsize[0]), in->textsize[1]);\n if (in->names && in->names->rowct > outlength){\n for (int k=outlength; k< in->names->rowct; k++)\n free(in->names->row[k]);\n in->names->rowct = outlength;\n }\n return in;\n}\n" }, { "path": "apop_db.m4.c", "content": "/** \\file apop_db.c\tAn easy front end to SQLite. Includes a few nice\nfeatures like a variance, skew, and kurtosis aggregator for SQL. */\n/* Copyright (c) 2006--2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n\n/** Here are where the options are initially set. See the \\ref apop_opts_type\n documentation for details.\n\\ingroup all_public\n*/\napop_opts_type apop_opts\t= \n { .verbose=1,\n .output_delimiter =\"\\t\", .input_delimiters = \"|,\\t\", \n .db_name_column = \"row_names\", .nan_string = \"NaN\", \n .db_engine = '\\0', .db_user = \"\\0\", \n .db_pass = \"\\0\", .stop_on_warning = 'n',\n .log_file = NULL,\n .rng_seed = 479901, .version = m4_apop_version };\n\n#define ERRCHECK {Apop_stopif(err, return 1, 0, \"%s: %s\",query, err); }\n#define ERRCHECK_NR {Apop_stopif(err, return NULL, 0, \"%s: %s\",query, err); }\n#define ERRCHECK_SET_ERROR(outdata) {Apop_stopif(err, if (!(outdata)) (outdata)=apop_data_alloc(); (outdata)->error='q'; sqlite3_free(err); return outdata, 0, \"%s: %s\",query, err); }\n\n#include \"apop_db_sqlite.c\" // callback_t is defined here, btw.\n\n\n#ifdef HAVE_MYSQL\n//Let mysql have these.\n#undef VERSION\n#undef PACKAGE\n#undef PACKAGE_NAME\n#undef PACKAGE_STRING\n#undef PACKAGE_TARNAME\n#undef PACKAGE_VERSION\n#undef PACKAGE_BUGREPORT\n#include \"apop_db_mysql.c\"\n#endif\n\n//if !apop_opts.db_engine, run this to assign a value.\nstatic void get_db_type(){\n if (getenv(\"APOP_DB_ENGINE\") && (!strcasecmp(getenv(\"APOP_DB_ENGINE\"), \"mysql\") || !strcasecmp(getenv(\"APOP_DB_ENGINE\"), \"mariadb\")))\n apop_opts.db_engine = 'm';\n else\n apop_opts.db_engine = 's';\n}\n\n//This macro declares the query string and fills it from the printf part of the call.\n#define Fillin(query, fmt) \\\n char *query; \\\n va_list argp; \\\n\tva_start(argp, fmt); \\\n\tApop_stopif(vasprintf(&query, fmt, argp)==-1, , 0, \"Trouble writing to a string.\"); \\\n\tva_end(argp); \\\n\tApop_notify(2, \"%s\", query);\n\n/** If you want to use a database on the hard drive instead of memory, then call this\nonce and only once before using any other database utilities.\n\nWith SQLite, if you want a disposable database which you won't use after the program\nends, don't bother with this function.\n\nThe trade-offs between an on-disk database and an in-memory db are as one would expect:\nmemory is faster, but the database is destroyed when the program exits.\n\nMySQL users: either set the environment variable APOP_DB_ENGINE=mysql or set \\c apop_opts.db_engine = 'm'.\n\nThe Apophenia package assumes you are only using a single database at a time. You\ncan use the SQL attach function to load other databases, or see this blog post for further\nsuggestions and sample code.\n\nWhen you are done doing your database manipulations, call \\ref apop_db_close if writing to disk.\n\n\\param filename\nThe name of a file on the hard drive on which to store the database. If\nNULL, then the database will be kept in memory (in which case,\nthe other database functions will call this function for you and you\ndon't need to bother).\n\n\\li See \\ref sqlsec for mroe notes on using databases.\n\n\\return 0: everything OK
\n 1: database did not open.\n*/\nint apop_db_open(char const *filename){\n if (!apop_opts.db_engine) get_db_type();\n if (!db) //check the environment.\n#ifdef HAVE_MYSQL\n if(!mysql_db) \n#endif\n\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n return apop_mysql_db_open(filename);\n#else\n {Apop_stopif(1, return -1, 0, \"Apophenia was compiled without mysql support.\");}\n#endif\n return apop_sqlite_db_open(filename);\n}\n\n/** \\cond doxy_ignore */\ntypedef struct {\n char const *name;\n int isthere;\n} tab_exists_t;\n/** \\endcond */\n\nstatic int tab_exists_callback(void *in, int argc, char **argv, char **whatever){\n tab_exists_t *te = in;\n\tif (!strcmp(argv[argc-1], te->name))\n\t\tte->isthere=1;\n\treturn 0;\n}\n\n/** Check for the existence of a table, and maybe delete it.\n\nRecreating a table which already exists can cause errors, so it is good practice to check for existence first. Also, this is the stylish way to delete a table, since just calling \"drop table\" will give you an error if the table doesn't exist.\n\n\\param name \tthe table name (no default)\n\\param remove 'd'\t==>delete table so it can be recreated in main.
\n\t\t'n'\t==>no action. Return result so program can continue. (default)\n\\return\n0 = table does not exist
\n1 = table was found, and if remove=='d', has been deleted\n-1 = processing error\n\n\\li In the SQLite engine, this function considers table views to be tables.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_table_exists(char const *name, char remove){\n char const *apop_varad_var(name, NULL)\n Apop_stopif(!name, return -1, 0, \"You gave me a NULL table name.\");\n char apop_varad_var(remove, 'n')\nAPOP_VAR_END_HEAD\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n return apop_mysql_table_exists(name, remove);\n#else\n Apop_stopif(1, return -1, 0, \"Apophenia was compiled without mysql support.\");\n#endif\n char *err=NULL, *q2;\n tab_exists_t te = { .name = name };\n tab_exists_t tev = { .name = name };\n\tif (db==NULL) return 0;\n\tsqlite3_exec(db, \"select name from sqlite_master where type='table'\", tab_exists_callback, &te, &err); \n\tsqlite3_exec(db, \"select name from sqlite_master where type='view'\", tab_exists_callback, &tev, &err); \n char query[]=\"Selecting names from sqlite_master\";//for ERRCHECK.\n\tERRCHECK\n\tif ((remove==1|| remove=='d') && (te.isthere||tev.isthere)){\n if (te.isthere)\n Asprintf(&q2, \"drop table %s;\", name);\n else\n Asprintf(&q2, \"drop view %s;\", name);\n\t\tsqlite3_exec(db, q2, NULL, NULL, &err); \n free(q2);\n ERRCHECK\n }\n\treturn (te.isthere||tev.isthere);\n}\n\n/**\nCloses the database on disk. If you opened the database with \\c apop_db_open(NULL), then this is basically optional.\n\n\\param vacuum \n'v': vacuum---do clean-up to minimize the size of the database on disk.
\n'q': Don't bother; just close the database. (default = 'q')\n\n\\return 0 on OK, nonzero on error.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_db_close(char vacuum){\n char apop_varad_var(vacuum, 'q')\nAPOP_VAR_END_HEAD\n if (apop_opts.db_engine == 'm') //assume this is set by now...\n#ifdef HAVE_MYSQL\n {apop_mysql_db_close(0);\n return 0;}\n#else\n {Apop_stopif(1, return -1, 0, \"Apophenia was compiled without mysql support.\");}\n#endif\n else {\n char *err, *query = \"db close\";//for errcheck.\n if (vacuum==1 || vacuum=='v') {\n sqlite3_exec(db, \"VACUUM\", NULL, NULL, &err);\n ERRCHECK\n }\n sqlite3_close(db);\n \t//ERRCHECK\n db = NULL;\n }\n return 0;\n}\n\n/** Send a query to the database that returns no data.\n\n\\li As with functions like the \\c apop_query_to_data, the query can include\nprintf-style format specifiers, such as apop_query(\"create table %s(id, name,\nage);\", tablename).\n\n\\param fmt A printf-style SQL query.\n\\return 0 on success, 1 on failure.\n*/\nint apop_query(const char *fmt, ...){\n char *err=NULL;\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n {Apop_stopif(!mysql_db, return 1, 0, \"No mySQL database is open.\");\n return apop_mysql_query(query);}\n#else\n Apop_stopif(1, return 1, 0, \"Apophenia was compiled without mysql support.\");\n#endif\n else \n {if (!db) apop_db_open(NULL);\n sqlite3_exec(db, query, NULL,NULL, &err);\n\t ERRCHECK\n }\n\tfree(query);\n\treturn 0;\n}\n\n/** Dump the results of a query into an array of strings.\n\n\\return\t An \\ref apop_data structure with the text element filled.\n\n\\param fmt A printf-style SQL query.\n\n\\exception out->error=='q' The database engine was unable to run the query (e.g., invalid SQL syntax). Again, a valid query that returns zero rows is not an error, and \\c NULL is returned.\n\\exception out->error=='d' Database error.\n\n\\li If apop_opts.db_name_column matches a column of the output table, then that\n column is used for row names, and therefore will not be included in the text.\n\\li query_output->text is always a 2-D array of strings, even if the query\n returns a single column. In that case, use returned_tab->text[i][0] (or\n equivalently, *returned_tab->text[i]) to refer to row i.\n\\li If an element in the database is \\c NULL, the corresponding cell in the output\n table will be filled with the text given by \\c apop_opts.nan_string. The default\n is \\c \"NaN\", but you can set apop_opts.nan_string = \"whatever you like\"\n to change the text to whatever you like.\n\\li Returns \\c NULL if your query is valid but returns zero rows.\n\\li The query can include printf-style format specifiers, such as\n apop_query_to_text(\"select name from %s where id=%i;\", tablename, id_number).\n\nFor example, the following function will list the tables in an SQLite database (much like you\ncould do from the command line using sqlite3 dbname.db \".table\").\n\n\\include ls_tables.c\n*/\napop_data * apop_query_to_text(const char * fmt, ...){\n apop_data *out = NULL;\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm'){\n#ifdef HAVE_MYSQL\n out = apop_mysql_query_core(query, process_result_set_chars);\n#else\n Apop_stopif(1, apop_return_data_error('d'), 0, \"Apophenia was compiled without mysql support.\");\n#endif\n } else out = apop_sqlite_query_to_text(query);\n free(query);\n return out;\n}\n\n//apop_query_to_data callback.\nstatic int db_to_table(void *qinfo, int argc, char **argv, char **column){\n Apop_stopif(!argv, return -1, apop_errorlevel, \"Got NULL data from SQLite.\");\n int i, ncfound = 0;\n callback_t *qi= qinfo;\n if (qi->firstcall){\n qi->firstcall--;\n for(i=0; inamecol = i;\n ncfound = 1;\n break;\n }\n\t qi->outdata = argc-ncfound ? apop_data_alloc(1, argc-ncfound) : apop_data_alloc( );\n for(i=0; inamecol != i)\n apop_name_add(qi->outdata->names, column[i], 'c');\n } else \n if (qi->outdata->matrix)\n apop_matrix_realloc(qi->outdata->matrix, qi->currentrow+1, qi->outdata->matrix->size2);\n ncfound =0;\n for (int jj=0;jjnamecol){\n double valor = \n !argv[jj] || !strcmp(argv[jj], \"NULL\")|| \n (apop_opts.nan_string && !strcasecmp(apop_opts.nan_string, argv[jj]))\n ? GSL_NAN : atof(argv[jj]);\n gsl_matrix_set(qi->outdata->matrix,qi->currentrow,jj-ncfound, valor);\n } else {\n apop_name_add(qi->outdata->names, argv[jj], 'r');\n ncfound = 1;\n }\n (qi->currentrow)++;\n\treturn 0;\n}\n\n/** Queries the database and dumps the result into an \\ref apop_data set.\n\n\\param fmt A printf-style SQL query.\n\n\\return If no rows are returned, \\c NULL; else an \\ref apop_data set with the data\nin place. Most data will be in the \\c matrix element of the output. Column names are\nappropriately placed. If \\ref apop_opts_type \"apop_opts.db_name_column\" matches one\nof the fields in your query's output (default: \\c row_names), then that column will\nbe used for row names (and therefore will not appear in the \\c matrix).\n\n\\exception out->error=='q' Query error. A valid query that returns no rows is not an error; in that case, you get \\c NULL.\n\n\\li The query can include printf-style\n format specifiers, such as apop_query_to_data(\"select age from %s where id=%i;\",\n tablename, id_number).\n\\li Blanks in the database (i.e., NULLs) and elements that match \\ref\n apop_opts_type \"apop_opts.nan_string\" are filled with NANs in the matrix.\n*/ \napop_data * apop_query_to_data(const char * fmt, ...){\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n return apop_mysql_query_core(query, process_result_set_data);\n#else\n Apop_stopif(1, apop_return_data_error('d'), 0, \"Apophenia was compiled without mysql support.\");\n#endif\n\n //else\n char *err=NULL;\n callback_t qinfo = {.firstcall = 1, .namecol=-1};\n\tif (db==NULL) apop_db_open(NULL);\n sqlite3_exec(db, query,db_to_table,&qinfo, &err); \n free (query);\n ERRCHECK_SET_ERROR(qinfo.outdata)\n\treturn qinfo.outdata;\n}\n\n\n /** \\cond doxy_ignore */\n//These used to do more, but I'll leave them as a macro anyway in case of future expansion.\n#define Store_settings \\\n int v = apop_opts.verbose; apop_opts.verbose=0;/*hack to prevent double-printing.*/ \\\n\n#define Restore_settings \\\n apop_opts.verbose=v;\n /** \\endcond */\n\n/** Queries the database and dumps the first column of the result into a \\c gsl_vector.\n\n\\param fmt A printf-style SQL query.\n\\return\t A gsl_vector holding the first column of the returned matrix. Thus, if your query returns multiple lines, you will get no warning, and the function will return the first in the list.\n\\exception out->error=='q' Query error. A valid query that returns no rows is not an error; in that case, you get \\c NULL.\n\n\\li Uses \\ref apop_query_to_data internally, then throws away all but the first column\n of the matrix.\n\\li If \\c apop_opts.db_name_column is set, then I'll ignore that column. It gets put\n into the names of the \\ref apop_data set, and then thrown away when I look at only\n the \\c gsl_matrix part of that set.\n\\li If the query returns zero rows of data or no columns, the function returns \\c NULL.\n\\li The query can include printf-style format specifiers, such as apop_query_to_vector(\"select age from %s where id=%i;\", tablename, id_number).\n*/\ngsl_vector * apop_query_to_vector(const char * fmt, ...){\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n return apop_mysql_query_core(query, process_result_set_vector);\n#else\n Apop_stopif(1, return NULL, 0, \"Apophenia was compiled without mysql support.\");\n#endif\n apop_data *d=NULL;\n gsl_vector *out;\n\tif (db==NULL) apop_db_open(NULL);\n Store_settings\n\td\t= apop_query_to_data(\"%s\", query);\n Restore_settings\n Apop_stopif(!d, return NULL, 2, \"Query [%s] turned up a blank table. Returning NULL.\", query);\n //else:\n out = gsl_vector_alloc(d->matrix->size1);\n\tgsl_matrix_get_col(out, d->matrix, 0);\n\tapop_data_free(d);\n free(query);\n\treturn out;\n}\n\n/** Queries the database, and dumps the result into a single double-precision floating point number.\n\n\\li This calls \\ref apop_query_to_data and returns the (0,0)th element of the returned matrix. Thus, if your query returns multiple lines, you will get no warning, and the function will return the first in the list (which is not always well-defined; maybe use an order by clause in your query if you expect multiple lines).\n\n\\li If \\c apop_opts.db_name_column is set, then I'll ignore that column. It gets put\n into the names of the \\ref apop_data set, and then thrown away when I look at only\n the \\c gsl_matrix element of that set.\n\\li If the query produces a blank table, returns \\c NAN, and if\n apop_opts.verbose>=2, prints an error.\n\\li The query can include printf-style format specifiers, such as\n apop_query_to_float(\"select age from %s where id=%i;\", tablename, id_number).\n\\li If the query produces an error, returns \\c NAN, and if apop_opts.verbose>=0,\n prints an error. If you need to distinguish between blank tables, NaNs in the data,\n and query errors, use \\ref apop_query_to_data.\n\n\\param fmt A printf-style SQL query.\n\\return\t\tA \\c double, actually.\n*/\ndouble apop_query_to_float(const char * fmt, ...){\n double out;\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm'){\n#ifdef HAVE_MYSQL\n out = apop_mysql_query_to_float(query);\n#else\n Apop_stopif(1, return NAN, 0, \"Apophenia was compiled without mysql support.\");\n#endif\n } else {\n apop_data *d=NULL;\n if (db==NULL) apop_db_open(NULL);\n Store_settings\n d = apop_query_to_data(\"%s\", query);\n Restore_settings\n Apop_stopif(!d, return GSL_NAN, 2, \"Query [%s] turned up a blank table. Returning NaN.\", query);\n Apop_stopif(d->error, return GSL_NAN, 0, \"Query [%s] failed. Returning NaN.\", query);\n out\t= apop_data_get(d);\n apop_data_free(d);\n }\n free(query);\n\treturn out;\n}\n\n/** Query data to an \\c apop_data set, but a mix of names, vectors, matrix elements, and text.\n\nIf you are querying to a matrix and maybe a name, use \\c\napop_query_to_data (and set \\ref apop_opts_type \"apop_opts.db_name_column\" if desired). If querying only text, use \\ref apop_query_to_text. But\nif your data is a mix of text and numbers, use this.\n\nThe first argument is a character string consisting of the letters \\c nvmtw, one for each column of the SQL output, indicating whether the column is a name, vector, matrix column, text column, or weight vector. You can have only one \\c n, one \\c v, and one \\c w. \n\nIf the query produces more columns than there are elements in the column specification, then the remainder are dumped into the text section. If there are fewer columns produced than given in the spec, the additional elements will be allocated but not filled (i.e., they are uninitialized and will have garbage).\n\n\n\\param typelist A string consisting of the letters \\c nvmtw. For example, if your query columns should go into a text column, the vector, the weights, and two matrix columns, this would be \"tvwmm\".\n\\param fmt A printf-style SQL query.\n\\exception out->error=='d' Dimension error. Your count of matrix parts didn't match what the query returned.\n\\exception out->error=='q' Query error. A valid query that returns no rows is not an error; in that case, you get \\c NULL.\n\n\\li \\ref apop_opts_type \"apop_opts.db_name_column\" is ignored. Use the \\c 'n' character\n to indicate the output column with row names.\n\\li As with the other \\c apop_query_to_... functions, the query can include printf-style\n format specifiers, such as apop_query_to_mixed_data(\"tv\", \"select name, age from\n\n*/\napop_data * apop_query_to_mixed_data(const char *typelist, const char * fmt, ...){\n Fillin(query, fmt)\n if (!apop_opts.db_engine) get_db_type();\n if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n {apop_data* out = apop_mysql_mixed_query(typelist, query);\n free(query);\n return out;}\n#else\n {Apop_notify(0, \"Apophenia was compiled without mysql support.\");\n return 0;}\n#endif\n //else\n apop_data *out = apop_sqlite_multiquery(typelist, query);\n free(query);\n return out;\n}\n\n/* Convenience function for extending a string. \n asprintf(%q, \"%s and stuff\", q);\n gives you a memory leak. This takes care of that.\n */\nvoid qxprintf(char **q, char *format, ...){\n va_list ap;\n char *r = *q;\n va_start(ap, format);\n Apop_stopif(vasprintf(q, format, ap)==-1, , 0, \"Trouble writing to a string.\");\n va_end(ap);\n free(r);\n}\n\nstatic void add_a_number (char **q, char *comma, double v){\n if (gsl_isnan(v))\n qxprintf(q,\"%s%c NULL \", *q, *comma);\n else if (isinf(v)==1)\n qxprintf(q,\"%s%c 'inf'\", *q, *comma);\n else if (isinf(v)==-1)\n qxprintf(q,\"%s%c '-inf' \", *q, *comma);\n else\n qxprintf(q,\"%s%c %g \",*q ,*comma, v);\n *comma = ',';\n}\n\nstatic int run_prepared_statements(apop_data const *set, sqlite3_stmt *p_stmt){\n#if SQLITE_VERSION_NUMBER < 3003009\n Apop_stopif(1, return -1, 0, \"Attempting to use prepared statements, but using a version of SQLite that doesn't support them.\");\n#else\n Get_vmsizes(set) //firstcol, msize1, maxsize\n for (size_t row=0; row < maxsize; row++){\n size_t field =1;\n if (set->names && set->names->rowct>row){\n if (!strlen(set->names->row[row])) field++; //leave NULL and cleared\n Apop_stopif(sqlite3_bind_text(p_stmt, field++, set->names->row[row], -1, SQLITE_TRANSIENT),\n return -1, apop_errorlevel, \n \"Something wrong with the row name for line %zu, [%s].\\n\" , row, set->names->row[row]);\n }\n if (set->vector && set->vector->size > row)\n Apop_stopif(sqlite3_bind_double(p_stmt, field++, apop_data_get(set, row, -1)),\n return -1, apop_errorlevel, \n \"Something wrong with the vector element on line %zu, [%g].\\n\" ,row, apop_data_get(set, row, -1));\n if (msize1 > row)\n for (size_t col=0; col < msize2; col++)\n Apop_stopif(sqlite3_bind_double(p_stmt, field++, apop_data_get(set, row, col)),\n return -1, apop_errorlevel, \n \"Something wrong with the matrix element %zu on line %zu, [%g].\\n\" ,col, row, apop_data_get(set, row, col));\n if (*set->textsize > row)\n for (size_t col=0; col < set->textsize[1]; col++){\n if (!strlen(set->text[row][col]) || (apop_opts.nan_string && !strcasecmp(apop_opts.nan_string, set->text[row][col])))\n {field++; continue;} //leave NULL and cleared\n Apop_stopif(sqlite3_bind_text(p_stmt, field++, set->text[row][col], -1, SQLITE_TRANSIENT),\n return -1, apop_errorlevel, \n \"Something wrong with a text element at row %zu, col %zu [%s].\\n\" , row, col, set->text[row][col]);\n }\n if (set->weights && set->weights->size > row)\n Apop_stopif(sqlite3_bind_double(p_stmt, field++, gsl_vector_get(set->weights, row)),\n return -1, apop_errorlevel, \n \"Something wrong with the weight element on line %zu, [%g].\\n\" ,row, gsl_vector_get(set->weights, row));\n int err = sqlite3_step(p_stmt);\n Apop_stopif(err!=0 && err != 101 //0=ok, 101=done\n , , 0, \"prepared sqlite insert query gave error code %i.\\n\", err);\n Apop_stopif(sqlite3_reset(p_stmt), return -1, apop_errorlevel, \"SQLite error.\");\n Apop_stopif(sqlite3_clear_bindings(p_stmt), return -1, apop_errorlevel, \"SQLite error.\"); //needed for NULLs\n }\n Apop_stopif(sqlite3_finalize(p_stmt)!=SQLITE_OK, return -1, apop_errorlevel, \"SQLite error.\");\n return 0;\n#endif\n}\n\n//users are expected to call apop_data_print.\nint apop_data_to_db(const apop_data *set, const char *tabname, const char output_append){\n Apop_stopif(!set, return -1, 1, \"you sent me a NULL data set. Database table %s will not be created.\", tabname);\n int\ti,j; \n char *q;\n char comma = ' ';\n int use_row = (apop_opts.db_name_column && strlen(apop_opts.db_name_column)) && set->names\n && ((set->matrix && set->names->rowct == set->matrix->size1)\n || (set->vector && set->names->rowct == set->vector->size));\n\n if (!apop_opts.db_engine) get_db_type();\n if (apop_table_exists(tabname))\n Asprintf(&q, \" \");\n else if (apop_opts.db_engine == 'm')\n#ifdef HAVE_MYSQL\n if (((output_append =='a' || output_append =='A') && apop_table_exists(tabname)))\n Asprintf(&q, \" \");\n else {\n Asprintf(&q, \"create table %s (\", tabname);\n if (use_row) {\n qxprintf(&q, \"%s\\n %s varchar(1000)\", q, apop_opts.db_name_column);\n comma = ',';\n }\n if (set->vector){\n if(!set->names || !set->names->vector) \n qxprintf(&q, \"%s%c\\n vector double \", q, comma);\n else\n qxprintf(&q, \"%s%c\\n %s double \", q,comma, set->names->vector);\n comma = ',';\n }\n if (set->matrix)\n for(i=0;i< set->matrix->size2; i++){\n if(!set->names || set->names->colct <= i) \n qxprintf(&q, \"%s%c\\n c%i double \", q, comma,i);\n else\n qxprintf(&q, \"%s%c\\n %s double \", q, comma, set->names->col[i]);\n comma = ',';\n }\n for(i=0;i< set->textsize[1]; i++){\n if (!set->names || set->names->textct <= i)\n qxprintf(&q, \"%s%c\\n tc%i varchar(1000) \", q, comma,i);\n else\n qxprintf(&q, \"%s%c\\n %s varchar(1000) \", q, comma, set->names->text[i]);\n comma = ',';\n }\n apop_query(\"%s); \", q);\n sprintf(q, \" \");\n }\n#else \n Apop_stopif(1, return -1, apop_errorlevel, \"Apophenia was compiled without mysql support.\");\n#endif\n else {\n if (db==NULL) apop_db_open(NULL);\n if (((output_append =='a' || output_append =='A') && apop_table_exists(tabname)) )\n Asprintf(&q, \" \");\n else {\n Asprintf(&q, \"create table %s (\", tabname);\n if (use_row) {\n qxprintf(&q, \"%s\\n %s\", q, apop_opts.db_name_column);\n comma = ',';\n }\n if (set->vector){\n if (!set->names || !set->names->vector) qxprintf(&q, \"%s%c\\n vector numeric\", q, comma);\n else qxprintf(&q, \"%s%c\\n \\\"%s\\\"\", q, comma, set->names->vector);\n comma = ',';\n }\n if (set->matrix)\n for(i=0;i< set->matrix->size2; i++){\n if(!set->names || set->names->colct <= i) \t\n qxprintf(&q, \"%s%c\\n c%i numeric\", q, comma,i);\n else\t\t\t\n qxprintf(&q, \"%s%c\\n \\\"%s\\\" numeric\", q, comma, set->names->col[i]);\n comma = ',';\n }\n for(i=0; i< set->textsize[1]; i++){\n if(!set->names || set->names->textct <= i) qxprintf(&q, \"%s%c\\n tc%i \", q, comma, i);\n else qxprintf(&q, \"%s%c\\n %s \", q, comma, set->names->text[i]);\n comma = ',';\n }\n if (set->weights) qxprintf(&q, \"%s%c\\n \\\"weights\\\" numeric\", q, comma);\n qxprintf(&q,\"%s);\",q);\n apop_query(\"%s\", q);\n qxprintf(&q,\" \");\n }\n }\n\n Get_vmsizes(set) //firstcol, msize2, maxsize\n int col_ct = (set->names ? !!set->names->rowct : 0) + set->textsize[1] + msize2 - firstcol + !!set->weights;\n Apop_stopif(!col_ct, return -1, 0, \"Input data set has zero columns of data (no rownames, text, matrix, vector, or weights). I can't create a table like that, sorry.\");\n if(apop_use_sqlite_prepared_statements(col_ct)){\n sqlite3_stmt *statement;\n Apop_stopif(\n apop_prepare_prepared_statements(tabname, col_ct, &statement), \n return -1, 0, \"Trouble preparing prepared statements.\");\n Apop_stopif(\n run_prepared_statements(set, statement), \n return -1, 0, \"error in insertions.\");\n } else {\n for(i=0; i< maxsize; i++){\n comma = ' ';\n qxprintf(&q, \"%s \\n insert into %s values(\",q, tabname);\n if (use_row){\n char *fixed= prep_string_for_sqlite(0, set->names->row[i]);\n qxprintf(&q, \"%s %s \",q, fixed);\n free(fixed);\n comma = ',';\n }\n if (set->vector)\n add_a_number (&q, &comma, gsl_vector_get(set->vector,i));\n if (set->matrix)\n for(j=0; j< set->matrix->size2; j++)\n add_a_number (&q, &comma, gsl_matrix_get(set->matrix,i,j));\n for(j=0; j< set->textsize[1]; j++){\n char *fixed= prep_string_for_sqlite(0, set->text[i][j]);\n qxprintf(&q, \"%s%c %s \",q, comma,fixed ? fixed : \"''\");\n free(fixed);\n comma = ',';\n }\n if (set->weights)\n add_a_number (&q, &comma, gsl_vector_get(set->weights,i));\n qxprintf(&q,\"%s);\",q);\n apop_query(\"%s\", q); \n q[0]='\\0';\n }\n }\n\tfree(q);\n return 0;\n}\n" }, { "path": "apop_db_mysql.c", "content": "/** \\file apop_db_mysql.c\nThis file is included directly into \\ref apop_db.c. It is read only if APOP_USE_MYSQL is defined.*/\n\n/* Copyright (c) 2006--2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \n#include \n\nstatic MYSQL *mysql_db; \n\n#define Areweconected(retval) Apop_stopif(!mysql_db, return retval, 0, \\\n \"No connection to a mySQL/mariadb database. apop_db_open() failure?\");\n\nstatic char *opt_host_name = NULL; /* server host (default=localhost) */\nstatic unsigned int opt_port_num = 0; /* port number (use built-in value) */\nstatic char *opt_socket_name = NULL; /* socket name (use built-in value) */\nstatic unsigned int opt_flags = 0; /* connection flags (none) */\n\n#define Apop_mstopif(cond, returnop, str) \\\n Apop_stopif(cond, returnop, 0, \\\n str \"\\n mySQL/mariadb error %u: %s\\n\", mysql_errno (mysql_db), mysql_error (mysql_db));\n\nstatic int apop_mysql_db_open(char const *in){\n Apop_stopif(!in, return 2, 0, \"MySQL needs a non-NULL db name.\");\n mysql_db = mysql_init (NULL);\n Apop_stopif(!mysql_db, return 1, 0, \"mysql_init() failed (probably out of memory)\");\n Apop_mstopif (!mysql_real_connect (mysql_db, opt_host_name, apop_opts.db_user, apop_opts.db_pass,\n in, opt_port_num, opt_socket_name, CLIENT_MULTI_STATEMENTS+opt_flags),\n mysql_close (mysql_db); return 1, \n \"mysql_real_connect() failed\");\n return 0;\n}\n\nstatic void apop_mysql_db_close(int ignoreme){\n if (mysql_db) mysql_close (mysql_db);\n}\n\n/*\n //Cut & pasted & cleaned from the mysql manual.\nstatic void process_results(void){\n else // mysql_store_result() returned nothing; should it have?\n Apop_stopif(mysql_field_count(mysql_db) == 0, , 0, \"apop_query error\");\n //else query wasn't a select & just didn't return data.\n}\n */\n\nstatic double apop_mysql_query(char *query){\n Apop_mstopif(mysql_query(mysql_db, query), return 1, \"apop_mysql_query failed\");\n MYSQL_RES *result = mysql_store_result(mysql_db);\n if (result) mysql_free_result(result);\n return 0;\n}\n\nstatic double apop_mysql_table_exists(char const *table, int delme){\n Areweconected(GSL_NAN);\n MYSQL_RES *res_set = mysql_list_tables(mysql_db, table);\n Apop_mstopif(!mysql_list_tables(mysql_db, table), return GSL_NAN,\n \"show tables query failed.\");\n int is_found = mysql_num_rows(res_set);\n mysql_free_result(res_set);\n if (!is_found) return 0;\n\n if (delme =='d' || delme=='D'){\n char *a_query;\n Asprintf(&a_query, \"drop table %s\", table);\n Apop_mstopif(mysql_query (mysql_db, a_query), GSL_NAN, \n \"table exists, but table dropping failed\");\n }\n return 1;\n}\n\n#define check_and_clean(do_if_failure) \\\n Apop_mstopif( mysql_errno (conn), \\\n if (out) do_if_failure; return NULL, \\\n \"mysql_fetch_row() failed\"); \\\n return out; \\\n\nstatic int get_name_row(unsigned int *num_fields, MYSQL_FIELD *fields){\n for(size_t i = 0; i < *num_fields; i++)\n if (apop_opts.db_name_column && !strcasecmp(fields[i].name, apop_opts.db_name_column)){\n (*num_fields)--;\n return i;\n }\n return -1;\n}\n\nstatic void * process_result_set_data (MYSQL *conn, MYSQL_RES *res_set) {\n MYSQL_ROW row;\n unsigned int num_fields = mysql_num_fields(res_set);\n unsigned int num_rows = mysql_num_rows (res_set);\n if (!num_fields || !num_rows) return NULL;\n\n MYSQL_FIELD *fields = mysql_fetch_fields(res_set);\n int name_row = get_name_row(&num_fields, fields);\n\n apop_data *out = apop_data_alloc(0, num_rows, num_fields);\n\n for(size_t i = 0; i < num_fields+ (name_row>=0); i++)\n if (i!=name_row) apop_name_add(out->names, fields[i].name, 'c');\n\n for (int i=0; (row = mysql_fetch_row (res_set)); i++) {\n int passed_name = 0;\n for (size_t j = 0; j < mysql_num_fields (res_set); j++){\n if (j==name_row){\n apop_name_add(out->names, row[j], 'r');\n passed_name = 1;\n continue;\n }\n if (!row[j]) apop_data_set(out, i , j-passed_name, NAN);\n else {\n char *end = NULL;\n double num = strtod(row[j], &end);\n apop_data_set(out, i , j-passed_name, *end ? NAN : num);\n }\n }\n }\n check_and_clean(apop_data_free(out))\n}\n\nstatic void * process_result_set_vector (MYSQL *conn, MYSQL_RES *res_set) {\n MYSQL_ROW row;\n unsigned int num_fields = mysql_num_fields(res_set);\n unsigned int num_rows = mysql_num_rows (res_set);\n if (num_fields == 0 || num_rows == 0) return NULL;\n gsl_vector *out = gsl_vector_alloc(num_rows);\n for (int j=0; (row = mysql_fetch_row (res_set)); j++){\n double valor = (!row[0] || !strcmp(row[0], \"NULL\"))\n ? GSL_NAN : atof(row[0]);\n gsl_vector_set(out, j, valor);\n }\n check_and_clean(gsl_vector_free(out))\n}\n\nstatic void * process_result_set_chars (MYSQL *conn, MYSQL_RES *res_set) {\n MYSQL_ROW row;\n unsigned int total_cols = mysql_num_fields(res_set);\n unsigned int total_rows = mysql_num_rows(res_set);\n\n MYSQL_FIELD *fields = mysql_fetch_fields(res_set);\n int name_row = get_name_row(&total_cols, fields);\n apop_data *out = apop_text_alloc(NULL, total_rows, total_cols);\n\n for (size_t i = 0; i < total_cols + (name_row>=0); i++)\n if (i!=name_row) apop_name_add(out->names, fields[i].name, 't');\n\n for (int i=0; (row = mysql_fetch_row (res_set)); i++){\n int passed_name = 0;\n\t\tfor (size_t jj=0; jjnames, row[jj], 'r');\n passed_name = 1;\n continue;\n }\n apop_text_set(out, i, jj-passed_name, \"%s\", (row[jj]==NULL)? apop_opts.nan_string : row[jj]);\n\t\t}\n }\n check_and_clean(;)\n}\n\nstatic void * apop_mysql_query_core(char *query, void *(*callback)(MYSQL*, MYSQL_RES*)){\n Areweconected(NULL);\n apop_data *output = NULL;\n Apop_mstopif(mysql_query (mysql_db, query), return NULL, \"mysql_query() failed\");\n MYSQL_RES *res_set = mysql_store_result (mysql_db);\n Apop_mstopif(!res_set, \n if (callback == process_result_set_data || callback==process_result_set_data) apop_return_data_error('q') \n else return NULL, \n \"mysql_store_result() failed\");\n if (!res_set->row_count) goto done; //just a blank table.\n output = callback(mysql_db, res_set);\n\n done:\n mysql_free_result (res_set);\n return output;\n}\n\nstatic double apop_mysql_query_to_float(char *query){\n Areweconected(GSL_NAN);\n Apop_mstopif(mysql_query (mysql_db, query) != 0, return GSL_NAN,\n \"mysql_query() failed\");\n MYSQL_RES *res_set = mysql_store_result (mysql_db);\n Apop_mstopif(!res_set, return GSL_NAN, \"mysql_store_result() failed\");\n if (mysql_num_rows(res_set)==0) return GSL_NAN;\n MYSQL_ROW row = mysql_fetch_row (res_set);\n Apop_mstopif(mysql_errno (mysql_db),\n mysql_free_result (res_set); return GSL_NAN,\n \"mysql_fetch_row() failed\");\n double out = atof(row[0]);\n mysql_free_result (res_set);\n return out;\n}\n\napop_data* apop_mysql_mixed_query(char const *intypes, char const *query){\n Areweconected(NULL);\n apop_data *out = NULL;\n Apop_mstopif(mysql_query (mysql_db, query), return NULL, \"mysql_query() failed\");\n MYSQL_RES *res_set = mysql_store_result(mysql_db);\n MYSQL_ROW row;\n Apop_mstopif(!res_set, return NULL, \"mysql_store_result() failed\");\n if (!res_set->row_count) goto done; //just a blank table.\n\n unsigned int total_cols = mysql_num_fields(res_set);\n unsigned int total_rows = mysql_num_rows(res_set);\n if (!total_cols || !total_rows) goto done;\n\n apop_qt info = { };\n count_types(&info, intypes); //in apop_db_sqlite.c\n //intypes[5] === names, vectors, mcols, textcols, weights.\n\n out = apop_data_alloc(info.intypes[1] ? total_rows : 0, \n info.intypes[2] ? total_rows : 0, \n info.intypes[2]);\n\n int requested = info.intypes[0]+info.intypes[1]+info.intypes[2]+info.intypes[3]+info.intypes[4];\n int excess = requested - total_cols;\n Apop_stopif(excess > 0, out->error='d' /*and continue.*/, 1, \n \"you asked for %i columns in your list of types(%s), but your query produced %u columns. \"\n \"The remainder will be placed in the text section. Output data set's ->error element set to 'd'.\" , requested, intypes, total_cols);\n Apop_stopif(excess < 0, out->error='d' /*and continue.*/, 1, \n \"you asked for %i columns in your list of types(%s), but your query produced %u columns. \"\n \"Ignoring the last %i type(s) in your list. Output data set's ->error element set to 'd'.\" , requested, intypes, total_cols, -excess);\n\n if (info.intypes[3]||excess>0) apop_text_alloc(out, total_rows, info.intypes[3] + ((excess > 0) ? excess : 0));\n if (info.intypes[4]) out->weights = gsl_vector_alloc(total_rows);\n\n MYSQL_FIELD *fields = mysql_fetch_fields(res_set);\n for (size_t i=0; inames, fields[i].name, 't');\n else if (c == 'v'|| c=='V')\n apop_name_add(out->names, fields[i].name, 'v');\n else if (c == 'm'|| c=='M')\n apop_name_add(out->names, fields[i].name, 'c');\n }\n\n for (int i=0; (row = mysql_fetch_row (res_set)); i++) {\n int thism=0, thist=0;\n\t\tfor (size_t j=0; jnames, row[j], 'r');\n else if (c == 't'|| c=='T')\n apop_text_set(out, i, thist++, \"%s\", (row[j]==NULL)? apop_opts.nan_string : row[j]);\n else if (c == 'v'|| c=='V'){\n double valor = (!row[j] || !strcmp(row[j], \"NULL\")) ? NAN : atof(row[j]);\n gsl_vector_set(out->vector, i, valor);\n } else if (c == 'w'|| c=='W'){\n double valor = (!row[j] || !strcmp(row[j], \"NULL\")) ? NAN : atof(row[j]);\n gsl_vector_set(out->weights, i, valor);\n } else if (c == 'm'|| c=='M')\n gsl_matrix_set(out->matrix, i , thism++, row[j] ? atof(row[j]): GSL_NAN);\n\t\t}\n }\n\n done:\n mysql_free_result (res_set);\n return out;\n}\n" }, { "path": "apop_db_sqlite.c", "content": "/** \\file apop_db_sqlite.c\nThis file is included directly into \\ref apop_db.c.\n\nCopyright (c) 2006--2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. \n */\n#include \n#include \n\nsqlite3\t*db=NULL;\t //There's only one SQLite database handle. Here it is.\n\n\n\n/** \\cond doxy_ignore */\ntypedef struct StdDevCtx StdDevCtx;\nstruct StdDevCtx {\n double avg; /* avg of terms */\n double avg2; /* avg of the squares of terms */\n double avg3; /* avg of the cube of terms */\n double avg4; /* avg of the fourth-power of terms */\n int cnt; /* Number of terms counted */\n};\n/** \\endcond */\n\nstatic void twoStep(sqlite3_context *context, int argc, sqlite3_value **argv){\n if (argc<1) return;\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if (p && argv[0]){\n double x = sqlite3_value_double(argv[0]);\n double ratio = p->cnt/(p->cnt+1.0);\n p->cnt++;\n p->avg\t*= ratio;\n p->avg2\t*= ratio;\n p->avg += x/(p->cnt +0.0);\n p->avg2 += gsl_pow_2(x)/(p->cnt +0.0);\n }\n}\n\nstatic void threeStep(sqlite3_context *context, int argc, sqlite3_value **argv){\n if (argc<1) return;\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if (p && argv[0]){\n double x = sqlite3_value_double(argv[0]);\n double ratio = p->cnt/(p->cnt+1.0);\n p->cnt++;\n p->avg\t*= ratio;\n p->avg2\t*= ratio;\n p->avg3\t*= ratio;\n p->avg += x/p->cnt;\n p->avg2 += gsl_pow_2(x)/p->cnt;\n p->avg3 += gsl_pow_3(x)/p->cnt;\n }\n}\n\nstatic void fourStep(sqlite3_context *context, int argc, sqlite3_value **argv){\n if( argc<1 ) return;\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if (p && argv[0]){\n double x = sqlite3_value_double(argv[0]);\n p->cnt++;\n p->avg = (x + p->avg * (p->cnt-1.))/p->cnt;\n p->avg2 = (gsl_pow_2(x)+ p->avg2 * (p->cnt-1.))/p->cnt;\n p->avg3 = (gsl_pow_3(x)+ p->avg3 * (p->cnt-1.))/p->cnt;\n p->avg4 = (gsl_pow_4(x)+ p->avg4 * (p->cnt-1.))/p->cnt;\n }\n}\n\nstatic void stdDevFinalizePop(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if (p && p->cnt>1)\n sqlite3_result_double(context, sqrt((p->avg2 - gsl_pow_2(p->avg))));\n else if (p->cnt == 1)\n \tsqlite3_result_double(context, 0);\n}\n\nstatic void varFinalizePop(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if( p && p->cnt>1 )\n sqlite3_result_double(context, (p->avg2 - gsl_pow_2(p->avg)));\n else if (p->cnt == 1)\n \tsqlite3_result_double(context, 0);\n}\n\nstatic void stdDevFinalize(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if( p && p->cnt>1 ){\n double rCnt = p->cnt;\n sqlite3_result_double(context,\n sqrt((p->avg2 - gsl_pow_2(p->avg))*rCnt/(rCnt-1.0)));\n } else if (p->cnt == 1)\n \tsqlite3_result_double(context, 0);\n}\n\nstatic void varFinalize(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if( p && p->cnt>1 ){\n double rCnt = p->cnt;\n sqlite3_result_double(context,\n (p->avg2 - gsl_pow_2(p->avg))*rCnt/(rCnt-1.0));\n } else if (p->cnt == 1)\n \tsqlite3_result_double(context, 0);\n}\n\nstatic void skewFinalize(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if( p && p->cnt>1 ){\n double rCnt = p->cnt;\n sqlite3_result_double(context,\n (p->avg3*rCnt - 3*p->avg2*p->avg*rCnt \n + 2*rCnt * gsl_pow_3(p->avg)) * rCnt/((rCnt-1.0)*(rCnt-2.0)));\n } else if (p->cnt == 1)\n \tsqlite3_result_double(context, 0);\n}\n\nstatic void kurtFinalize(sqlite3_context *context){\n StdDevCtx *p = sqlite3_aggregate_context(context, sizeof(*p));\n if( p && p->cnt>1 ){\n double n = p->cnt;\n double kurtovern = p->avg4 - 4*p->avg3*p->avg\n + 6 * p->avg2*gsl_pow_2(p->avg)\n - 3* gsl_pow_4(p->avg);\n double var = p->avg2 - gsl_pow_2(p->avg);\n long double coeff0= n*n/(gsl_pow_3(n)*(gsl_pow_2(n)-3*n+3));\n long double coeff1= n*gsl_pow_2(n-1)+ (6*n-9);\n long double coeff2= n*(6*n-9);\n sqlite3_result_double(context, coeff0*(coeff1 * kurtovern + coeff2 * gsl_pow_2(var)));\n } else if (p->cnt == 1)\n sqlite3_result_double(context, 0);\n}\n\nstatic void powFn(sqlite3_context *context, int argc, sqlite3_value **argv){\n double base = sqlite3_value_double(argv[0]);\n double exp = sqlite3_value_double(argv[1]);\n sqlite3_result_double(context, pow(base, exp));\n}\n\nstatic void rngFn(sqlite3_context *context, int argc, sqlite3_value **argv){\n Staticdef(gsl_rng *, rng, apop_rng_alloc(apop_opts.rng_seed++));\n //sqlite3_result_double(context, gsl_rng_uniform(rng));\n sqlite3_result_double(context, gsl_rng_uniform(apop_rng_get_thread(-1)));\n}\n\n#define sqfn(name) static void name##Fn(sqlite3_context *context, int argc, sqlite3_value **argv){ \\\n sqlite3_result_double(context, name(sqlite3_value_double(argv[0]))); }\n\nsqfn(sqrt) sqfn(exp) sqfn(log) sqfn(log10) sqfn(sin) \nsqfn(cos) sqfn(tan) sqfn(asin) sqfn(acos) sqfn(atan)\n\n\nstatic int apop_sqlite_db_open(char const *filename){\n int status = sqlite3_open(filename ? filename : \":memory:\", &db);\n Apop_stopif(status, db=NULL; return status,\n 0, \"The database %s didn't open.\", filename ? filename : \"in memory\");\n\tsqlite3_create_function(db, \"stddev\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &stdDevFinalize);\n\tsqlite3_create_function(db, \"std\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &stdDevFinalizePop);\n\tsqlite3_create_function(db, \"stddev_samp\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &stdDevFinalize);\n\tsqlite3_create_function(db, \"stddev_pop\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &stdDevFinalizePop);\n\tsqlite3_create_function(db, \"var\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &varFinalize);\n\tsqlite3_create_function(db, \"var_samp\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &varFinalize);\n\tsqlite3_create_function(db, \"var_pop\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &varFinalizePop);\n\tsqlite3_create_function(db, \"variance\", 1, SQLITE_ANY, NULL, NULL, &twoStep, &varFinalizePop);\n\tsqlite3_create_function(db, \"skew\", 1, SQLITE_ANY, NULL, NULL, &threeStep, &skewFinalize);\n\tsqlite3_create_function(db, \"kurt\", 1, SQLITE_ANY, NULL, NULL, &fourStep, &kurtFinalize);\n\tsqlite3_create_function(db, \"kurtosis\", 1, SQLITE_ANY, NULL, NULL, &fourStep, &kurtFinalize);\n\tsqlite3_create_function(db, \"ln\", 1, SQLITE_ANY, NULL, &logFn, NULL, NULL);\n\tsqlite3_create_function(db, \"ran\", 0, SQLITE_ANY, NULL, &rngFn, NULL, NULL);\n\tsqlite3_create_function(db, \"pow\", 2, SQLITE_ANY, NULL, &powFn, NULL, NULL);\n\n#define sqlink(name) sqlite3_create_function(db, #name , 1, SQLITE_ANY, NULL, &name##Fn, NULL, NULL);\n sqlink(sqrt) sqlink(exp) sqlink(sin) sqlink(cos)\n sqlink(tan) sqlink(asin) sqlink(acos) sqlink(atan) sqlink(log) sqlink(log10)\n\tapop_query(\"pragma short_column_names\");\n return 0;\n}\n\n/** \\cond doxy_ignore */\ntypedef struct { //for the apop_query_to_... functions.\n int firstcall, namecol;\n size_t currentrow;\n apop_data *outdata;\n} callback_t;\n/** \\endcond */\n\n//This is the callback for apop_query_to_text.\nstatic int db_to_chars(void *qinfo,int argc, char **argv, char **column){\n callback_t *qi= qinfo;\n apop_data* d = qi->outdata; //alias. Allocated in calling fn.\n int\taddnames = 0, ncshift=0;\n if (!d->names->textct) addnames++;\n if (qi->firstcall){\n qi->firstcall = 0;\n for(int i=0; inamecol = i;\n break;\n }\n }\n int rows = d->textsize[0];\n int cols = argc - (qi->namecol >= 0);\n apop_text_alloc(d, rows+1, cols);//doesn't move d.\n for (size_t jj=0; jjnamecol){\n apop_name_add(d->names, argv[jj], 'r'); \n ncshift ++;\n } else {\n apop_text_set(d, rows, jj-ncshift, (argv[jj]==NULL)? apop_opts.nan_string: argv[jj]);\n //Asprintf(&(d->text[rows][jj-ncshift]), \"%s\", (argv[jj]==NULL)? \"NaN\": argv[jj]);\n if(addnames)\n apop_name_add(d->names, column[jj], 't'); \n }\n return 0;\n}\n\napop_data * apop_sqlite_query_to_text(char *query){\n char *err = NULL;\n callback_t qinfo = {.outdata=apop_data_alloc(), .namecol=-1, .firstcall=1};\n if (db==NULL) apop_db_open(NULL);\n sqlite3_exec(db, query, db_to_chars, &qinfo, &err); ERRCHECK_SET_ERROR(qinfo.outdata)\n if (qinfo.outdata->textsize[0]==0){\n apop_data_free(qinfo.outdata);\n return NULL;\n }\n return qinfo.outdata;\n}\n\n/** \\cond doxy_ignore */\ntypedef struct {\n apop_data *d;\n int intypes[5];//names, vectors, mcols, textcols, weights.\n int current, thisrow, error_thrown;\n const char *instring;\n} apop_qt;\n/** \\endcond */\n\nstatic void count_types(apop_qt *in, const char *intypes){\n int i = 0;\n char c;\n in->instring = intypes;\n while ((c=intypes[i++]))\n if (c=='n'||c=='N') in->intypes[0]++;\n else if (c=='v'||c=='V') in->intypes[1]++;\n else if (c=='m'||c=='M') in->intypes[2]++;\n else if (c=='t'||c=='T') in->intypes[3]++;\n else if (c=='w'||c=='W') in->intypes[4]++;\n if (in->intypes[0]>1)\n Apop_notify(1, \"You asked apop_query_to_mixed data for multiple row names. I'll ignore all but the last one.\");\n if (in->intypes[1]>1)\n Apop_notify(1, \"You asked apop_query_to_mixed for multiple vectors. I'll ignore all but the last one.\");\n if (in->intypes[4]>1)\n Apop_notify(1, \"You asked apop_query_to_mixed for multiple weighting vectors. I'll ignore all but the last one.\");\n}\n\nstatic int multiquery_callback(void *instruct, int argc, char **argv, char **column){\n apop_qt *in = instruct;\n char c;\n int thistcol = 0, \n thismcol = 0,\n colct = 0,\n i, addnames = 0;\n in->thisrow ++;\n if (!in->d) {\n in->d = in->intypes[2]\n ? apop_data_alloc(in->intypes[1], 1, in->intypes[2])\n : apop_data_alloc(in->intypes[1]);\n if (in->intypes[4])\n in->d->weights = gsl_vector_alloc(1);\n if (in->intypes[3]){\n in->d->textsize[0] = 1;\n in->d->textsize[1] = in->intypes[3];\n in->d->text = malloc(sizeof(char***));\n }\n }\n if (!(in->d->names->colct + in->d->names->textct + (in->d->names->vector!=NULL)))\n addnames++;\n if (in->d->textsize[1]){\n in->d->textsize[0] = in->thisrow;\n in->d->text = realloc(in->d->text, sizeof(char ***)*in->thisrow);\n in->d->text[in->thisrow-1] = malloc(sizeof(char**) * in->d->textsize[1]);\n }\n if (in->intypes[2])\n apop_matrix_realloc(in->d->matrix, in->thisrow, in->intypes[2]);\n for (i=in->current=0; i< argc; i++){\n c = in->instring[in->current++];\n if (c=='n'||c=='N'){\n apop_name_add(in->d->names, (argv[i]? argv[i] : \"NaN\") , 'r'); \n if(addnames)\n apop_name_add(in->d->names, column[i], 'h'); \n } else if (c=='v'||c=='V'){\n apop_vector_realloc(in->d->vector, in->thisrow);\n apop_data_set(in->d, in->thisrow-1, -1, \n argv[i] ? atof(argv[i]) : GSL_NAN);\n if(addnames)\n apop_name_add(in->d->names, column[i], 'v'); \n } else if (c=='m'||c=='M'){\n apop_data_set(in->d, in->thisrow-1, thismcol++, \n argv[i] ? atof(argv[i]) : GSL_NAN);\n if(addnames)\n apop_name_add(in->d->names, column[i], 'c'); \n } else if (c=='t'||c=='T'){\n Asprintf(&(in->d->text[in->thisrow-1][thistcol++]), \"%s\", \n\t\t\t argv[i] ? argv[i] : \"NaN\");\n if(addnames)\n apop_name_add(in->d->names, column[i], 't'); \n } else if (c=='w'||c=='W'){\n apop_vector_realloc(in->d->weights, in->thisrow);\n gsl_vector_set(in->d->weights, in->thisrow-1, \n argv[i] ? atof(argv[i]) : GSL_NAN);\n }\n colct++;\n }\n int requested = in->intypes[0]+in->intypes[1]+in->intypes[2]+in->intypes[3]+in->intypes[4];\n Apop_stopif(colct != requested, in->error_thrown='d'; return 1, 1, \n \"you asked for %i columns in your list of types(%s), but your query produced %u columns. \"\n \"The remainder will be placed in the text section. Output data set's ->error element set to 'd'.\" , requested, in->instring, colct);\n return 0;\n}\n\napop_data *apop_sqlite_multiquery(const char *intypes, char *query){\n Apop_stopif(!intypes, apop_return_data_error('t'), 0, \"You gave me NULL for the list of input types. I can't work with that.\");\n Apop_stopif(!query, apop_return_data_error('q'), 0, \"You gave me a NULL query. I can't work with that.\");\n char *err = NULL;\n apop_qt info = { };\n count_types(&info, intypes);\n\tif (!db) apop_db_open(NULL);\n sqlite3_exec(db, query, multiquery_callback, &info, &err); \n Apop_stopif(info.error_thrown, if (!info.d) apop_data_alloc(); info.d->error='d'; return info.d,\n 0, \"dimension error\");\n ERRCHECK_SET_ERROR(info.d)\n\treturn info.d;\n}\n" }, { "path": "apop_fexact.c", "content": "/** \\file apop_fexact.c\n\n Fisher's exact test for contingency tables \n\n This file primarily consists of an algorithm from the ACM, fully\n documented below. The C code below was cut and pasted from the\n R project. Thanks, guys.\n\nUn-R-ifying modifications Copyright (c) 2006--2009 by Ben Klemens.\nLicensed under the GPLv2; see COPYING.\n\nR version credits:\nfexact.f -- translated by f2c (version 19971204).\\\\\nRun through a slightly modified version of MM's f2c-clean.\\\\\nHeavily hand-edited by KH and MM.\n */\n\n#include \"apop_internal.h\"\n#include \n#include \n#include \n\n/* These are the R-specific items. */\ntypedef enum { FALSE = 0, TRUE /*, MAYBE */ } Rboolean;\nint imax2(int a, int b){return (a>b) ? a : b;}\nint imin2(int a, int b){return (ab) ? a : b;}\nfloat fmin2(float a, float b){return (a MULT is now an __argument__ of the function\n 3. FEXACT may be converted to single precision by setting IREAL = 3,\n and converting all DOUBLE PRECISION specifications (except the\n specifications for RWRK, IWRK, and DWRK) to REAL.\tThis will\n require changing the names and specifications of the intrinsic\n functions ALOG, AMAX1, AMIN1, EXP, and REAL. In addition, the\n machine specific constants will need to be changed, and the name\n DWRK will need to be changed to RWRK in the call to F2XACT.\n 4. Machine specific constants are specified and documented in F2XACT.\n A missing value code is specified in both FEXACT and F2XACT.\n 5. Although not a restriction, is is not generally practical to call\n this routine with large tables which are not sparse and in\n which the 'hybrid' algorithm has little effect. For example,\n although it is feasible to compute exact probabilities for the\n table\n\t 1 8 5 4 4 2 2\n\t 5 3 3 4 3 1 0\n\t 10 1 4 0 0 0 0,\n computing exact probabilities for a similar table which has been\n enlarged by the addition of an extra row (or column) may not be\n feasible.\n -----------------------------------------------------------------------\n */\n\n /* To increase the length of the table of past path lengths relative\n to the length of the hash table, increase MULT.\n */\n\n /* AMISS is a missing value indicator which is returned when the\n probability is not defined.\n */\n const double amiss = GSL_NAN;\n /*\n Set IREAL = 4 for DOUBLE PRECISION\n Set IREAL = 3 for SINGLE PRECISION\n */\n#define i_real 4\n#define i_int 2\n\n /* System generated locals */\n int ikh;\n /* Local variables */\n int nco, nro, ntot, numb, iiwk, irwk;\n int i, j, k, kk, ldkey, ldstp, i1, i2, i3, i4, i5, i6, i7, i8, i9, i10;\n int i3a, i3b, i3c, i9a, iwkmax, iwkpt;\n\n /* Workspace Allocation */\n double *equiv;\n iwkmax = 2 * (int) (*workspace / 2);\n equiv = (double *) calloc(iwkmax / 2, sizeof(double));\n\n#define dwrk (equiv)\n#define iwrk ((int *)equiv)\n#define rwrk ((float *)equiv)\n\n iwkpt = 0;\n\n Apop_stopif(*nrow > *ldtabl, has_error=true; return, 0, \"NROW must be less than or equal to LDTABL.\");\n\n ntot = 0;\n for (i = 0; i < *nrow; ++i) {\n for (j = 0; j < *ncol; ++j) {\n if (table[i + j * *ldtabl] < 0)\n prterr(2, \"All elements of TABLE may not be negative.\");\n ntot += table[i + j * *ldtabl];\n }\n }\n if (ntot == 0) {\n prterr(3, \"All elements of TABLE are zero.\\n\"\n \"PRT and PRE are set to missing values.\");\n *pre = *prt = amiss;\n free(equiv);\n return;\n }\n\n /* nco := max(*nrow, *ncol)\n * nro := min(*nrow, *ncol) */\n if(*ncol > *nrow) {\n nco = *ncol;\n nro = *nrow;\n } else {\n nco = *nrow;\n nro = *ncol;\n }\n k = *nrow + *ncol + 1;\n kk = k * nco;\n\n ikh = ntot + 1;\n i1 = iwork(iwkmax, &iwkpt, ikh, i_real);\n i2 = iwork(iwkmax, &iwkpt, nco, i_int);\n i3 = iwork(iwkmax, &iwkpt, nco, i_int);\n i3a = iwork(iwkmax, &iwkpt, nco, i_int);\n i3b = iwork(iwkmax, &iwkpt, nro, i_int);\n i3c = iwork(iwkmax, &iwkpt, nro, i_int);\n ikh = imax2(k * 5 + (kk << 1), nco * 7 + 800);\n iiwk= iwork(iwkmax, &iwkpt, ikh, i_int);\n ikh = imax2(nco + 401, k);\n irwk= iwork(iwkmax, &iwkpt, ikh, i_real);\n\n /* NOTE:\n What follows below splits the remaining amount iwkmax - iwkpt of\n (int) workspace into hash tables as follows.\n type size index\n\t INT 2 * ldkey i4 i5 i11\n\t REAL 2 * ldkey i8 i9 i10\n\t REAL 2 * ldstp i6\n\t INT 6 * ldstp i7\n Hence, we need ldkey times\n 3 * 2 + 3 * 2 * s + 2 * mult * s + 6 * mult\n chunks of integer memory, where s = sizeof(REAL) / sizeof(INT).\n If doubles are used and are twice as long as ints, this gives\n 18 + 10 * mult\n so that the value of ldkey can be obtained by dividing available\n (int) workspace by this number.\n\n In fact, because iwork() can actually s * n + s - 1 int chunks\n when allocating a REAL, we use ldkey = available / numb - 1.\n\n FIXME:\n Can we always assume that sizeof(double) / sizeof(int) is 2?\n */\n\n if (i_real == 4) \t\t/* Double precision reals */\n numb = 18 + 10 * *mult;\n else\t\t\t/* Single precision reals */\n numb = (*mult << 3) + 12;\n ldkey = (iwkmax - iwkpt) / numb - 1;\n ldstp = *mult * ldkey;\n ikh = ldkey << 1;\ti4 = iwork(iwkmax, &iwkpt, ikh, i_int);\n ikh = ldkey << 1;\ti5 = iwork(iwkmax, &iwkpt, ikh, i_int);\n ikh = ldstp << 1;\ti6 = iwork(iwkmax, &iwkpt, ikh, i_real);\n ikh = ldstp * 6;\ti7 = iwork(iwkmax, &iwkpt, ikh, i_int);\n ikh = ldkey << 1;\ti8 = iwork(iwkmax, &iwkpt, ikh, i_real);\n ikh = ldkey << 1;\ti9 = iwork(iwkmax, &iwkpt, ikh, i_real);\n ikh = ldkey << 1;\ti9a = iwork(iwkmax, &iwkpt, ikh, i_real);\n ikh = ldkey << 1;\ti10 = iwork(iwkmax, &iwkpt, ikh, i_int);\n\n /* To convert to double precision, change RWRK to DWRK in the next CALL.\n */\n f2xact(*nrow,\n\t *ncol,\n\t table,\n\t *ldtabl,\n\t expect,\n\t percnt,\n\t emin,\n\t prt,\n\t pre,\n\t dwrk + i1,\n\t iwrk + i2,\n\t iwrk + i3,\n\t iwrk + i3a,\n\t iwrk + i3b,\n\t iwrk + i3c,\n\t iwrk + i4,\n\t &ldkey,\n\t iwrk + i5,\n\t dwrk + i6,\n\t &ldstp,\n\t iwrk + i7,\n\t dwrk + i8,\n\t dwrk + i9,\n\t dwrk + i9a,\n\t iwrk + i10,\n\t iwrk + iiwk,\n\t dwrk + irwk);\n\n free(equiv);\n}\n\n#undef rwrk\n#undef iwrk\n#undef dwrk\n\nstatic void f2xact(int nrow, int ncol, int *table, int ldtabl,\n double *expect, double *percnt, double *emin, double *prt,\n double *pre, double *fact, int *ico, int *iro, int *kyy,\n int *idif, int *irn, int *key, int *ldkey, int *ipoin,\n double *stp, int *ldstp, int *ifrq, double *LP, double *SP,\n double *tm, int *key2, int *iwk, double *rwk) {\n/* -----------------------------------------------------------------------\n Name:\t\tF2XACT\n Purpose:\tComputes Fisher's exact test for a contingency table,\n\t\troutine with workspace variables specified.\n -----------------------------------------------------------------------\n */\n const int imax = INT_MAX;/* the largest representable int on the machine.*/\n\n /* AMISS is a missing value indicator which is returned when the\n probability is not defined. */\n const double amiss = GSL_NAN;\n\n /* TOL is chosen as the square root of the smallest relative spacing. */\n const static double tol = 3.45254e-7;\n\n const char* ch_err_5 =\n\t\"The hash table key cannot be computed because the largest key\\n\"\n\t\"is larger than the largest representable int.\\n\"\n\t\"The algorithm cannot proceed.\\n\"\n\t\"Reduce the workspace size or use another algorithm.\";\n\n /* Local variables -- changed from \"static\"\n * (*does* change results very slightly on i386 linux) */\n int i, ii, j, k, n,\n\tiflag,ifreq, ikkey, ikstp, ikstp2, ipn, ipo, itop, itp = 0,\n\tjkey, jstp, jstp2, jstp3, jstp4, k1, kb, kd, ks, kval = 0, kmax, last,\n\tncell, ntot, nco, nro, nro2, nrb,\n\ti31, i32, i33, i34, i35, i36, i37, i38, i39,\n\ti41, i42, i43, i44, i45, i46, i47, i48, i310, i311;\n\n double dspt, d1,dd,df,ddf, drn,dro, obs, obs2, obs3, pastp,pv, tmp=0.;\n\n#ifndef USING_R\n double d2;\n int ifault;\n#endif\n Rboolean nr_gt_nc, maybe_chisq, chisq = FALSE/* -Wall */, psh;\n\n /* Parameter adjustments */\n table -= ldtabl + 1;\n --ico;\n --iro;\n --kyy;\n --idif;\n --irn;\n\n --key;\n --ipoin;\n --stp;\n --ifrq;\n --LP;\n --SP;\n --tm;\n --key2;\n --iwk;\n --rwk;\n\n /* Check table dimensions */\n Apop_stopif(nrow > ldtabl, has_error=true; return, 0, \"NROW must be less than or equal to LDTABL.\");\n Apop_stopif(ncol <= 1, has_error=true; return, 0, \"NCOL must be at least 2\");\n\n /* Initialize KEY array */\n for (i = 1; i <= *ldkey << 1; ++i) {\n key[i] = -9999;\n key2[i] = -9999;\n }\n\n nr_gt_nc = nrow > ncol;\n /* nco := max(nrow, ncol) : */\n if (nr_gt_nc) nco = nrow;\n else nco = ncol;\n /* Compute row marginals and total */\n ntot = 0;\n for (i = 1; i <= nrow; ++i) {\n iro[i] = 0;\n for (j = 1; j <= ncol; ++j) {\n Apop_stopif(table[i + j * ldtabl] < 0., has_error=true; return,\n 0, \"All elements of TABLE must be non-negative.\");\n iro[i] += table[i + j * ldtabl];\n }\n ntot += iro[i];\n }\n\n if (ntot == 0) {\n prterr(3, \"All elements of TABLE are zero.\\n\"\n \"PRT and PRE are set to missing values.\");\n *pre = *prt = amiss;\n return;\n }\n\n /* Column marginals */\n for (i = 1; i <= ncol; ++i) {\n ico[i] = 0;\n for (j = 1; j <= nrow; ++j)\n ico[i] += table[j + i * ldtabl];\n }\n\n /* sort marginals */\n isort(&nrow, &iro[1]);\n isort(&ncol, &ico[1]);\n\n /*\tDetermine row and column marginals.\n\tDefine max(nrow,ncol) =: nco >= nro := min(nrow,ncol)\n\tnco is defined above\n\n\tSwap marginals if necessary to\tico[1:nco] & iro[1:nro]\n */\n if (nr_gt_nc) {\n nro = ncol;\n /* Swap marginals */\n for (i = 1; i <= nco; ++i) {\n ii = iro[i];\n if (i <= nro) iro[i] = ico[i];\n ico[i] = ii;\n }\n } else\n nro = nrow;\n\n /* Get multiplers for stack */\n kyy[1] = 1;\n for (i = 1; i < nro; ++i) {\n /* Hash table multipliers */\n if (iro[i] + 1 <= imax / kyy[i]) {\n kyy[i + 1] = kyy[i] * (iro[i] + 1);\n j /= kyy[i];\n }\n else {\n prterr(5, ch_err_5);\n return;\n }\n }\n\n /* Check for Maximum product : */\n /* original code: if (iro[nro - 1] + 1 > imax / kyy[nro - 1]) */\n if (iro[nro] + 1 > imax / kyy[nro]) {\n prterr(501, ch_err_5);\n return;\n }\n\n /* Compute log factorials */\n fact[0] = 0.;\n fact[1] = 0.;\n if(ntot >= 2) fact[2] = log(2.);\n /* MM: old code assuming log() to be SLOW */\n for (i = 3; i <= ntot; i += 2) {\n fact[i] = fact[i - 1] + log((double) i);\n j = i + 1;\n if (j <= ntot)\n fact[j] = fact[i] + fact[2] + fact[j / 2] - fact[j / 2 - 1];\n }\n /* Compute obs := observed path length */\n obs = tol;\n ntot = 0;\n for (j = 1; j <= nco; ++j) {\n dd = 0.;\n if (nr_gt_nc) {\n for (i = 1; i <= nro; ++i) {\n dd += fact[table[j + i * ldtabl]];\n ntot += table[j + i * ldtabl];\n }\n } else {\n for (i = 1, ii = j * ldtabl + 1; i <= nro; i++, ii++) {\n dd += fact[table[ii]];\n ntot += table[ii];\n }\n }\n obs += fact[ico[j]] - dd;\n }\n\n /* Denominator of observed table: DRO */\n dro = f9xact(nro, ntot, &iro[1], fact);\n /* improve: the following \"easily\" underflows to zero -- return \"log()\" */\n *prt = exp(obs - dro);\n *pre = 0.;\n itop = 0;\n maybe_chisq = (*expect > 0.);\n\n /* Initialize pointers for workspace */\n /* f3xact */\n i31 = 1;\n i32 = i31 + nco;\n i33 = i32 + nco;\n i34 = i33 + nco;\n i35 = i34 + nco;\n i36 = i35 + nco;\n i37 = i36 + nco;\n i38 = i37 + nco;\n i39 = i38 + 400;\n i310 = 1;\n i311 = 1 + 400;\n /* f4xact */\n i = nrow + ncol + 1;\n i41 = 1;\n i42 = i41 + i;\n i43 = i42 + i;\n i44 = i43 + i;\n i45 = i44 + i;\n i46 = i45 + i;\n i47 = i46 + i * nco;\n i48 = 1;\n\n /* Initialize pointers */\n k = nco;\n last = *ldkey + 1;\n jkey = *ldkey + 1;\n jstp = *ldstp + 1;\n jstp2 = *ldstp * 3 + 1;\n jstp3 = (*ldstp << 2) + 1;\n jstp4 = *ldstp * 5 + 1;\n ikkey = 0;\n ikstp = 0;\n ikstp2 = *ldstp << 1;\n ipo = 1;\n ipoin[1] = 1;\n stp[1] = 0.;\n ifrq[1] = 1;\n ifrq[ikstp2 + 1] = -1;\n\nOuter_Loop:\n kb = nco - k + 1;\n ks = 0;\n n = ico[kb];\n kd = nro + 1;\n kmax = nro;\n /* IDIF is the difference in going to the daughter */\n for (i = 1; i <= nro; ++i)\n idif[i] = 0;\n\n /* Generate the first daughter */\n do {\n --kd;\n ntot = imin2(n, iro[kd]);\n idif[kd] = ntot;\n if (idif[kmax] == 0)\n --kmax;\n n -= ntot;\n } while (n > 0 && kd != 1);\n\n if (n != 0) /* i.e. kd == 1 */\n\tgoto L310;\n\n k1 = k - 1;\n n = ico[kb];\n ntot = 0;\n for (i = kb + 1; i <= nco; ++i)\n ntot += ico[i];\n\nL150:\n /* Arc to daughter length=ICO[KB] */\n for (i = 1; i <= nro; ++i)\n irn[i] = iro[i] - idif[i];\n\n if (k1 > 1) {\n /* Sort irn */\n if (nro == 2) {\n if (irn[1] > irn[2]) {\n ii = irn[1]; irn[1] = irn[2]; irn[2] = ii;\n }\n } else\n isort(&nro, &irn[1]);\n\n /* Adjust start for zero */\n for (i = 1; i <= nro; ++i) {\n if (irn[i] != 0)\n break;\n }\n nrb = i;\n } else \n nrb = 1;\n nro2 = nro - nrb + 1;\n\n /* Some table values */\n ddf = f9xact(nro, n, &idif[1], fact);\n drn = f9xact(nro2, ntot, &irn[nrb], fact) - dro + ddf;\n /* Get hash value */\n if (k1 > 1) {\n kval = irn[1];\n /* Note that with the corrected check at error \"502\",\n * we won't have overflow in kval below : */\n for (i = 2; i <= nro; ++i)\n kval += irn[i] * kyy[i];\n\n /* Get hash table entry */\n i = kval % (*ldkey << 1) + 1;\n /* Search for unused location */\n for (itp = i; itp <= *ldkey << 1; ++itp) {\n ii = key2[itp];\n if (ii == kval) {\n goto L240;\n } else if (ii < 0) {\n key2[itp] = kval;\n LP[itp] = 1.;\n SP[itp] = 1.;\n goto L240;\n }\n }\n\n for (itp = 1; itp <= i - 1; ++itp) {\n ii = key2[itp];\n if (ii == kval) \n goto L240;\n else if (ii < 0) {\n key2[itp] = kval;\n LP[itp] = 1.;\n goto L240;\n }\n }\n\n /* KH\n prterr(6, \"LDKEY is too small.\\n\"\n \"It is not possible to give the value of LDKEY required,\\n\"\n \"but you could try doubling LDKEY (and possibly LDSTP).\");\n */\n prterr(6, \"LDKEY is too small for this problem.\\n\"\n \"Try increasing the size of the workspace.\");\n }\n\nL240:\n psh = TRUE;\n /* Recover pastp */\n ipn = ipoin[ipo + ikkey];\n pastp = stp[ipn + ikstp];\n ifreq = ifrq[ipn + ikstp];\n /* Compute shortest and longest path */\n if (k1 > 1) {\n obs2 = obs - fact[ico[kb + 1]] - fact[ico[kb + 2]] - ddf;\n for (i = 3; i <= k1; ++i)\n obs2 -= fact[ico[kb + i]];\n\n if (LP[itp] > 0.) {\n dspt = obs - obs2 - ddf;\n /* Compute longest path */\n LP[itp] = f3xact(nro2, &irn[nrb], k1, &ico[kb + 1], ntot, fact,\n &iwk[i31], &iwk[i32], &iwk[i33], &iwk[i34],\n &iwk[i35], &iwk[i36], &iwk[i37], &iwk[i38],\n &iwk[i39], &rwk[i310], &rwk[i311], &tol);\n if(LP[itp] > 0.) {/* can this happen? */\n printf(\"___ LP[itp=%d] = %g > 0\\n\", itp, LP[itp]);\n LP[itp] = 0.;\n }\n\n /* Compute shortest path -- using dspt as offset */\n SP[itp] = f4xact(nro2, &irn[nrb], k1, &ico[kb + 1], dspt, fact,\n &iwk[i47], &iwk[i41], &iwk[i42], &iwk[i43],\n &iwk[i44], &iwk[i45], &iwk[i46], &rwk[i48], &tol);\n /* SP[itp] = fmin2(0., SP[itp] - dspt);*/\n if(SP[itp] > 0.) { /* can this happen? */\n printf(\"___ SP[itp=%d] = %g > 0\\n\", itp, SP[itp]);\n SP[itp] = 0.;\n }\n\n /* Use chi-squared approximation? */\n if (maybe_chisq && (irn[nrb] * ico[kb + 1]) > ntot * *emin) {\n ncell = 0.;\n for (i = 0; i < nro2; ++i)\n for (j = 1; j <= k1; ++j)\n if (irn[nrb + i] * ico[kb + j] >= ntot * *expect)\n ncell++;\n\n if (ncell * 100 >= k1 * nro2 * *percnt) {\n tmp = 0.;\n for (i = 0; i < nro2; ++i)\n tmp += (fact[irn[nrb + i]] -\n fact[irn[nrb + i] - 1]);\n tmp *= k1 - 1;\n for (j = 1; j <= k1; ++j)\n tmp += (nro2 - 1) * (fact[ico[kb + j]] -\n fact[ico[kb + j] - 1]);\n df = (double) ((nro2 - 1) * (k1 - 1));\n tmp += df * 1.83787706640934548356065947281;\n tmp -= (nro2 * k1 - 1) * (fact[ntot] - fact[ntot - 1]);\n tm[itp] = (obs - dro) * -2. - tmp;\n } else {\n /* tm[itp] set to a flag value */\n tm[itp] = -9876.;\n }\n } else\n tm[itp] = -9876.;\n }\n obs3 = obs2 - LP[itp];\n obs2 -= SP[itp];\n if (tm[itp] == -9876.) \n chisq = FALSE;\n else {\n chisq = TRUE;\n tmp = tm[itp];\n }\n } else {\n obs2 = obs - drn - dro;\n obs3 = obs2;\n }\n\nL300:\n /* Process node with new PASTP */\n if (pastp <= obs3) /* Update pre */\n *pre += (double) ifreq * exp(pastp + drn);\n else if (pastp < obs2) {\n if (chisq) {\n df = (double) ((nro2 - 1) * (k1 - 1));\n d1 = fmax2(0., tmp + (pastp + drn) * 2.) / 2.;\n d2 = df / 2.;\n pv = 1. - gammds(&d1, &d2, &ifault);\n *pre += (double) ifreq * exp(pastp + drn) * pv;\n } else {\n /* Put daughter on queue */\n d1 = pastp + ddf;\n f5xact(&d1, &tol, &kval, &key[jkey], ldkey, &ipoin[jkey],\n &stp[jstp], ldstp, &ifrq[jstp], &ifrq[jstp2],\n &ifrq[jstp3], &ifrq[jstp4], &ifreq, &itop, psh);\n psh = FALSE;\n }\n }\n /* Get next PASTP on chain */\n ipn = ifrq[ipn + ikstp2];\n if (ipn > 0) {\n pastp = stp[ipn + ikstp];\n ifreq = ifrq[ipn + ikstp];\n goto L300;\n }\n /* Generate a new daughter node */\n f7xact(kmax, &iro[1], &idif[1], &kd, &ks, &iflag);\n if (iflag != 1)\n\tgoto L150;\n\n\nL310:\n /* Go get a new mother from stage K */\n do {\n if(!f6xact(nro, &iro[1], &kyy[1], &key[ikkey + 1], ldkey, &last, &ipo))\n /* Update pointers */\n goto Outer_Loop;\n\n /* else : no additional nodes to process */\n --k;\n itop = 0;\n ikkey = jkey - 1;\n ikstp = jstp - 1;\n ikstp2 = jstp2 - 1;\n jkey = *ldkey - jkey + 2;\n jstp = *ldstp - jstp + 2;\n jstp2 = (*ldstp << 1) + jstp;\n for (i = 1; i <= *ldkey << 1; ++i)\n key2[i] = -9999;\n } while (k >= 2);\n}/* f2xact() */\n\nstatic double f3xact(int nrow, int *irow, int ncol, int *icol,\n int ntot, double *fact, int *ico, int *iro, int *it,\n int *lb, int *nr, int *nt, int *nu, int *itc, int *ist,\n double *stv, double *alen, const double *tol) {\n/* -----------------------------------------------------------------------\n Name:\t F3XACT\n Purpose: Computes the longest path length for a given table.\n\n Arguments:\n NROW - The number of rows in the table.\t\t\t(Input)\n IROW - Vector of length NROW containing the row sums\n for the table.\t\t\t\t\t(Input)\n NCOL - The number of columns in the table.\t\t(Input)\n ICOL - Vector of length K containing the column sums\n for the table.\t\t\t\t\t(Input)\n NTOT - The total count in the table.\t\t\t(Input)\n FACT - Vector containing the logarithms of factorials.\t(Input)\n ICO - Work vector of length MAX(NROW,NCOL).\n IRO - Work vector of length MAX(NROW,NCOL).\n IT\t - Work vector of length MAX(NROW,NCOL).\n LB\t - Work vector of length MAX(NROW,NCOL).\n NR\t - Work vector of length MAX(NROW,NCOL).\n NT\t - Work vector of length MAX(NROW,NCOL).\n NU\t - Work vector of length MAX(NROW,NCOL).\n ITC - Work vector of length 400.\n IST - Work vector of length 400.\n STV - Work vector of length 400.\n ALEN - Work vector of length MAX(NROW,NCOL).\n TOL - Tolerance.\t\t\t\t\t(Input)\n\n Return Value :\n LP - The longest path for the table.\t\t\t(Output)\n -----------------------------------------------------------------------\n */\n\n const int ldst = 200;/* half stack size */\n /* Initialized data */\n static int nst = 0;\n static int nitc = 0;\n\n int i, k;\n int n11, n12, ii, nn, ks, ic1, ic2, nc1, nn1;\n int nr1, nco, nct, ipn, irl, key, lev, itp, nro, nrt, kyy, nc1s;\n double LP, v, val, vmn;\n Rboolean xmin;\n\n --stv;\n --ist;\n --itc;\n --nu;\n --nt;\n --nr;\n --lb;\n --it;\n --iro;\n --ico;\n --icol;\n --irow;\n\n if (nrow <= 1) {\t/* nrow is 1 */\n LP = 0.;\n if (nrow > 0) \n for (i = 1; i <= ncol; ++i)\n LP -= fact[icol[i]];\n return LP;\n }\n\n if (ncol <= 1) {\t/* ncol is 1 */\n LP = 0.;\n if (ncol > 0) {\n for (i = 1; i <= nrow; ++i)\n LP -= fact[irow[i]];\n }\n return LP;\n }\n\n /* 2 by 2 table */\n if (nrow * ncol == 4) {\n n11 = (irow[1] + 1) * (icol[1] + 1) / (ntot + 2);\n n12 = irow[1] - n11;\n return -(fact[n11] + fact[n12] +\n fact[icol[1] - n11] + fact[icol[2] - n12]);\n }\n\n /* ELSE: larger than 2 x 2 : */\n\n /* Test for optimal table */\n val = 0.;\n if (irow[nrow] <= irow[1] + ncol) \n xmin = f10act(nrow, &irow[1], ncol, &icol[1], &val, fact,\n\t\t &lb[1], &nu[1], &nr[1]);\n else xmin = FALSE;\n if (! xmin && icol[ncol] <= icol[1] + nrow) \n xmin = f10act(ncol, &icol[1], nrow, &irow[1], &val, fact,\n\t\t &lb[1], &nu[1], &nr[1]);\n if (xmin)\n return - val;\n\n /* Setup for dynamic programming */\n\n for (i = 0; i <= ncol; ++i)\n alen[i] = 0.;\n for (i = 1; i <= 2*ldst; ++i)\n ist[i] = -1;\n\n nn = ntot;\n /* Minimize ncol */\n if (nrow >= ncol) {\n nro = nrow;\n nco = ncol;\n ico[1] = icol[1];\n nt[1] = nn - ico[1];\n for (i = 2; i <= ncol; ++i) {\n ico[i] = icol[i];\n nt[i] = nt[i - 1] - ico[i];\n }\n for (i = 1; i <= nrow; ++i)\n iro[i] = irow[i];\n } else {\n nro = ncol;\n nco = nrow;\n ico[1] = irow[1];\n nt[1] = nn - ico[1];\n for (i = 2; i <= nrow; ++i) {\n ico[i] = irow[i];\n nt[i] = nt[i - 1] - ico[i];\n }\n for (i = 1; i <= ncol; ++i)\n iro[i] = icol[i];\n }\n\n nc1s = nco - 1;\n kyy = ico[nco] + 1;\n /* Initialize pointers */\n vmn = 1e100;/* to contain min(v..) */\n irl = 1;\n ks = 0;\n k = ldst;\n\nLnewNode: /* Setup to generate new node */\n\n lev = 1;\n nr1 = nro - 1;\n nrt = iro[irl];\n nct = ico[1];\n lb[1] = (int) ((((double) nrt + 1) * (nct + 1)) /\n\t\t (double) (nn + nr1 * nc1s + 1) - *tol) - 1;\n nu[1] = (int) ((((double) nrt + nc1s) * (nct + nr1)) /\n\t\t (double) (nn + nr1 + nc1s)) - lb[1] + 1;\n nr[1] = nrt - lb[1];\n\nLoopNode: /* Generate a node */\n --nu[lev];\n if (nu[lev] == 0) {\n\tif (lev == 1)\n\t goto L200;\n\n\t--lev;\n\tgoto LoopNode;\n }\n ++lb[lev];\n --nr[lev];\n\n while(1) {\n alen[lev] = alen[lev - 1] + fact[lb[lev]];\n if (lev >= nc1s)\n break;\n\n nn1 = nt[lev];\n nrt = nr[lev];\n ++lev;\n nc1 = nco - lev;\n nct = ico[lev];\n lb[lev] = (int) ((((double) nrt + 1) * (nct + 1)) /\n (double) (nn1 + nr1 * nc1 + 1) - *tol);\n nu[lev] = (int) ((((double) nrt + nc1) * (nct + nr1)) /\n (double) (nn1 + nr1 + nc1) - lb[lev] + 1);\n nr[lev] = nrt - lb[lev];\n }\n alen[nco] = alen[lev] + fact[nr[lev]];\n lb[nco] = nr[lev];\n\n v = val + alen[nco];\n\n if (nro == 2) { /* Only 1 row left */\n v += fact[ico[1] - lb[1]] + fact[ico[2] - lb[2]];\n for (i = 3; i <= nco; ++i)\n v += fact[ico[i] - lb[i]];\n\n if (v < vmn)\n vmn = v;\n } else if (nro == 3 && nco == 2) { /* 3 rows and 2 columns */\n\tnn1 = nn - iro[irl] + 2;\n\tic1 = ico[1] - lb[1];\n\tic2 = ico[2] - lb[2];\n\tn11 = (iro[irl + 1] + 1) * (ic1 + 1) / nn1;\n\tn12 = iro[irl + 1] - n11;\n\tv += fact[n11] + fact[n12] + fact[ic1 - n11] + fact[ic2 - n12];\n\tif (v < vmn)\n\t vmn = v;\n\n } else { /* Column marginals are new node */\n\n\tfor (i = 1; i <= nco; ++i)\n\t it[i] = imax2(ico[i] - lb[i], 0);\n\t\n\t/* Sort column marginals it[] : */\n\tif (nco == 2) {\n\t if (it[1] > it[2]) { /* swap */\n\t\tii = it[1]; it[1] = it[2]; it[2] = ii;\n\t }\n\t} else\n\t isort(&nco, &it[1]);\n\n\t/* Compute hash value */\n\tkey = it[1] * kyy + it[2];\n\tfor (i = 3; i <= nco; ++i) {\n\t key = it[i] + key * kyy;\n\t}\n\tif (key < -1){\n if (apop_opts.verbose)\n\t printf(\"Bug in FEXACT: gave negative key.\\n\");\n return -1;\n }\n\t/* Table index */\n\tipn = key % ldst + 1;\n\t/* Find empty position */\n\tfor (itp = ipn, ii = ks + ipn; itp <= ldst; ++itp, ++ii) {\n\t if (ist[ii] < 0) {\n\t\tgoto L180;\n\t } else if (ist[ii] == key) {\n\t\tgoto L190;\n\t }\n\t}\n\n\tfor (itp = 1, ii = ks + 1; itp <= ipn - 1; ++itp, ++ii) {\n\t if (ist[ii] < 0) {\n\t\tgoto L180;\n\t } else if (ist[ii] == key) {\n\t\tgoto L190;\n\t }\n\t}\n\n\t/* this happens less, now that we check for negative key above: */\n\tApop_stopif(1, has_error=true; return GSL_NAN, 0, \"Stack length exceeded in f3xact. This problem should not occur.\");\n\nL180: /* Push onto stack */\n\tist[ii] = key;\n\tstv[ii] = v;\n\t++nst;\n\tii = nst + ks;\n\titc[ii] = itp;\n\tgoto LoopNode;\n\nL190: /* Marginals already on stack */\n\tstv[ii] = fmin2(v, stv[ii]);\n }\n goto LoopNode;\n\n\nL200: /* Pop item from stack */\n if (nitc > 0) {\n\t/* Stack index */\n\titp = itc[nitc + k] + k;\n\t--nitc;\n\tval = stv[itp];\n\tkey = ist[itp];\n\tist[itp] = -1;\n\t/* Compute marginals */\n\tfor (i = nco; i >= 2; --i) {\n\t ico[i] = key % kyy;\n\t key /= kyy;\n\t}\n\tico[1] = key;\n\t/* Set up nt array */\n\tnt[1] = nn - ico[1];\n\tfor (i = 2; i <= nco; ++i)\n\t nt[i] = nt[i - 1] - ico[i];\n\n\t/* Test for optimality (L90) */\n\tif (iro[nro] <= iro[irl] + nco) {\n\t xmin = f10act(nro, &iro[irl], nco, &ico[1], &val, fact,\n\t\t\t &lb[1], &nu[1], &nr[1]);\n\t} else xmin = FALSE;\n\n\tif (!xmin && ico[nco] <= ico[1] + nro)\n\t xmin = f10act(nco, &ico[1], nro, &iro[irl], &val, fact,\n\t\t\t &lb[1], &nu[1], &nr[1]);\n\tif (xmin) {\n\t if (vmn > val)\n\t\tvmn = val;\n\t goto L200;\n\t}\n\telse goto LnewNode;\n\n } else if (nro > 2 && nst > 0) {\n /* Go to next level */\n nitc = nst;\n nst = 0;\n k = ks;\n ks = ldst - ks;\n nn -= iro[irl];\n ++irl;\n --nro;\n goto L200;\n }\n return - vmn;\n}\n\nstatic double f4xact(int nrow, int *irow, int ncol, int *icol, double dspt,\n double *fact, int *icstk, int *ncstk, int *lstk, int *mstk,\n int *nstk, int *nrstk, int *irstk, double *ystk, const double *tol) {\n/* -----------------------------------------------------------------------\n Name:\t F4XACT\n Purpose: Computes the shortest path length for a given table.\n\n Arguments:\n NROW - The number of rows in the table.\t(Input)\n IROW - Vector of length NROW containing the row sums for the\n\t table. (Input)\n NCOL - The number of columns in the table. (Input)\n ICOL - Vector of length K containing the column sums for the\n\t table. (Input)\n DSPT - \"offset\" for SP computation\n FACT - Vector containing the logarithms of factorials. (Input)\n ICSTK - NCOL by NROW+NCOL+1 work array.\n NCSTK - Work vector of length NROW+NCOL+1.\n LSTK - Work vector of length NROW+NCOL+1.\n MSTK - Work vector of length NROW+NCOL+1.\n NSTK - Work vector of length NROW+NCOL+1.\n NRSTK - Work vector of length NROW+NCOL+1.\n IRSTK - NROW by MAX(NROW,NCOL) work array.\n YSTK - Work vector of length NROW+NCOL+1.\n TOL - Tolerance.\t\t\t\t\t(Input)\n\n Return Value :\n\n SP\t - The shortest path for the table.\t\t\t(Output)\n ----------------------------------------------------------------------- */\n\n int i, j, k, l, m, n, ic1, ir1, ict, irt, istk, nco, nro;\n double y, amx, SP;\n\n /* Take care of the easy cases first */\n if (nrow == 1) {\n SP = 0.;\n for (i = 0; i < ncol; ++i)\n SP -= fact[icol[i]];\n return SP;\n }\n if (ncol == 1) {\n SP = 0.;\n for (i = 0; i < nrow; ++i)\n SP -= fact[irow[i]];\n return SP;\n }\n if (nrow * ncol == 4) {\n if (irow[1] <= icol[1])\n return -(fact[irow[1]] + fact[icol[1]] + fact[icol[1] - irow[1]]);\n else\n return -(fact[icol[1]] + fact[irow[1]] + fact[irow[1] - icol[1]]);\n }\n\n /* Parameter adjustments */\n irstk -= nrow + 1;\n icstk -= ncol + 1;\n\n --nrstk;\n --ncstk;\n --lstk;\n --mstk;\n --nstk;\n --ystk;\n\n /* initialization before loop */\n for (i = 1; i <= nrow; ++i)\n irstk[i + nrow] = irow[nrow - i];\n\n for (j = 1; j <= ncol; ++j)\n icstk[j + ncol] = icol[ncol - j];\n\n nro = nrow;\n nco = ncol;\n nrstk[1] = nro;\n ncstk[1] = nco;\n ystk[1] = 0.;\n y = 0.;\n istk = 1;\n l = 1;\n amx = 0.;\n SP = dspt;\n\n /* First LOOP */\n do {\n\tir1 = irstk[istk * nrow + 1];\n\tic1 = icstk[istk * ncol + 1];\n\tif (ir1 > ic1) {\n\t if (nro >= nco) {\n m = nco - 1;\tn = 2;\n\t } else {\n m = nro;\tn = 1;\n\t }\n\t} else if (ir1 < ic1) {\n\t if (nro <= nco) {\n m = nro - 1;\tn = 1;\n\t } else {\n m = nco;\tn = 2;\n\t }\n\t} else {\n\t if (nro <= nco) {\n m = nro - 1;\tn = 1;\n\t } else {\n m = nco - 1;\tn = 2;\n\t }\n\t}\n\n L60:\n\tif (n == 1) {\n\t i = l; j = 1;\n\t} else {\n\t i = 1; j = l;\n\t}\n\n\tirt = irstk[i + istk * nrow];\n\tict = icstk[j + istk * ncol];\n\ty += fact[imin2(irt, ict)];\n\tif (irt == ict) {\n\t --nro;\n\t --nco;\n\t f11act(&irstk[istk * nrow + 1], i, nro,\n\t\t &irstk[(istk + 1) * nrow + 1]);\n\t f11act(&icstk[istk * ncol + 1], j, nco,\n\t\t &icstk[(istk + 1) * ncol + 1]);\n\t} else if (irt > ict) {\n\t --nco;\n\t f11act(&icstk[istk * ncol + 1], j, nco,\n\t\t &icstk[(istk + 1) * ncol + 1]);\n\t f8xact(&irstk[istk * nrow + 1], irt - ict, i, nro,\n\t\t &irstk[(istk + 1) * nrow + 1]);\n\t} else {\n\t --nro;\n\t f11act(&irstk[istk * nrow + 1], i, nro,\n\t\t &irstk[(istk + 1) * nrow + 1]);\n\t f8xact(&icstk[istk * ncol + 1], ict - irt, j, nco,\n\t\t &icstk[(istk + 1) * ncol + 1]);\n\t}\n\n\tif (nro == 1) {\n\t for (k = 1; k <= nco; ++k)\n y += fact[icstk[k + (istk + 1) * ncol]];\n\t break;\n\t}\n\tif (nco == 1) {\n\t for (k = 1; k <= nro; ++k)\n y += fact[irstk[k + (istk + 1) * nrow]];\n\t break;\n\t}\n\n\tlstk[istk] = l;\n\tmstk[istk] = m;\n\tnstk[istk] = n;\n\t++istk;\n\tnrstk[istk] = nro;\n\tncstk[istk] = nco;\n\tystk[istk] = y;\n\tl = 1;\n } while(1);/* end do */\n\n if (y > amx) {\n amx = y;\n if (SP - amx <= *tol)\n return -dspt;\n }\n\n do {\n\t--istk;\n\tif (istk == 0) {\n\t SP -= amx;\n\t if (SP - amx <= *tol)\n\t\treturn -dspt;\n\t else\n\t\treturn SP - dspt;\n\t}\n\tl = lstk[istk] + 1;\n\n\tfor(;; ++l) {\n\t if (l > mstk[istk])\tbreak;\n\n\t n = nstk[istk];\n\t nro = nrstk[istk];\n\t nco = ncstk[istk];\n\t y = ystk[istk];\n\t if (n == 1) {\n\t\tif (irstk[l\t+ istk * nrow] <\n\t\t irstk[l - 1 + istk * nrow])\tgoto L60;\n\t }\n\t else if (n == 2) {\n\t\tif (icstk[l\t+ istk * ncol] <\n\t\t icstk[l - 1 + istk * ncol])\tgoto L60;\n\t }\n\t}\n } while(1);\n}\n\n\nvoid f5xact(double *pastp, const double *tol, int *kval, int *key, int *ldkey,\n int *ipoin, double *stp, int *ldstp, int *ifrq, int *npoin,\n int *nr, int *nl, int *ifreq, int *itop, Rboolean psh) {\n/* -----------------------------------------------------------------------\n Name:\t F5XACT aka \"PUT\"\n Purpose: Put node on stack in network algorithm.\n\n Arguments:\n PASTP - The past path length.\t\t\t\t(Input)\n TOL - Tolerance for equivalence of past path lengths. \t(Input)\n KVAL - Key value. \t\t\t\t\t(Input)\n KEY - Vector of length LDKEY containing the key values.\t(in/out)\n LDKEY - Length of vector KEY. \t\t\t\t(Input)\n IPOIN - Vector of length LDKEY pointing to the\n\t linked list of past path lengths. \t\t(in/out)\n STP - Vector of length LSDTP containing the\n\t linked lists of past path lengths. \t\t(in/out)\n LDSTP - Length of vector STP. \t\t\t\t(Input)\n IFRQ - Vector of length LDSTP containing the past path\n\t frequencies. \t\t\t\t\t(in/out)\n NPOIN - Vector of length LDSTP containing the pointers to\n\t the next past path length. \t\t\t(in/out)\n NR\t - Vector of length LDSTP containing the right object\n\t pointers in the tree of past path lengths. (in/out)\n NL\t - Vector of length LDSTP containing the left object\n\t pointers in the tree of past path lengths. (in/out)\n IFREQ - Frequency of the current path length. (Input)\n ITOP - Pointer to the top of STP. \t\t\t(Input)\n PSH - Logical.\t\t \t\t\t\t(Input)\n\t If PSH is true, the past path length is found in the\n\t table KEY. Otherwise the location of the past path\n\t length is assumed known and to have been found in\n\t a previous call. ==>>>>> USING \"static\" variables\n ----------------------------------------------------------------------- */\n\n static int itmp, ird, ipn, itp; /* << *need* static, see PSH above */\n double test1, test2;\n\n --nl;\n --nr;\n --npoin;\n --ifrq;\n --stp;\n\n /* Function Body */\n if (psh) {\n\t/* Convert KVAL to int in range 1, ..., LDKEY. */\n\tird = *kval % *ldkey;\n\t/* Search for an unused location */\n\tfor (itp = ird; itp < *ldkey; ++itp) {\n\t if (key[itp] == *kval)\n\t\tgoto L40;\n\n\t if (key[itp] < 0)\n\t\tgoto L30;\n\t}\n\tfor (itp = 0; itp < ird; ++itp) {\n\t if (key[itp] == *kval)\n\t\tgoto L40;\n\n\t if (key[itp] < 0)\n\t\tgoto L30;\n\t}\n\t/* Return if KEY array is full */\n\t/* KH\n\t prterr(6, \"LDKEY is too small for this problem.\\n\"\n\t \"It is not possible to estimate the value of LDKEY \"\n\t \"required,\\n\"\n\t \"but twice the current value may be sufficient.\");\n\t */\n\tprterr(6, \"LDKEY is too small for this problem.\\n\"\n\t \"Try increasing the size of the workspace.\");\n\n\nL30: /* Update KEY */\n\n\tkey[itp] = *kval;\n\t++(*itop);\n\tipoin[itp] = *itop;\n\t/* Return if STP array full */\n\tif (*itop > *ldstp) {\n\t /* KH\n\t prterr(7, \"LDSTP is too small for this problem.\\n\"\n\t \"It is not possible to estimate the value of LDSTP \"\n\t \"required,\\n\"\n\t \"but twice the current value may be sufficient.\");\n\t */\n\t prterr(7, \"LDSTP is too small for this problem.\\n\"\n\t\t \"Try increasing the size of the workspace.\");\n\t}\n\t/* Update STP, etc. */\n\tnpoin[*itop] = -1;\n\tnr [*itop] = -1;\n\tnl [*itop] = -1;\n\tstp [*itop] = *pastp;\n\tifrq [*itop] = *ifreq;\n\treturn;\n }\n\nL40: /* Find location, if any, of pastp */\n\n ipn = ipoin[itp];\n test1 = *pastp - *tol;\n test2 = *pastp + *tol;\n\n do {\n\tif (stp[ipn] < test1)\n\t ipn = nl[ipn];\n\telse if (stp[ipn] > test2)\n\t ipn = nr[ipn];\n\telse {\n\t ifrq[ipn] += *ifreq;\n\t return;\n\t}\n } while (ipn > 0);\n\n /* Return if STP array full */\n ++(*itop);\n if (*itop > *ldstp) {\n\t/*\n\t prterr(7, \"LDSTP is too small for this problem.\\n\"\n\t \"It is not possible to estimate the value of LDSTP \"\n\t \"required,\\n\"\n\t \"but twice the current value may be sufficient.\");\n\t */\n\tprterr(7, \"LDSTP is too small for this problem.\\n\"\n\t \"Try increasing the size of the workspace.\");\n\treturn;\n }\n\n /* Find location to add value */\n ipn = ipoin[itp];\n itmp = ipn;\n\nL60:\n if (stp[ipn] < test1) {\n\titmp = ipn;\n\tipn = nl[ipn];\n\tif (ipn > 0)\n\t goto L60;\n\t/* else */\n\tnl[itmp] = *itop;\n }\n else if (stp[ipn] > test2) {\n\titmp = ipn;\n\tipn = nr[ipn];\n\tif (ipn > 0)\n\t goto L60;\n\t/* else */\n\tnr[itmp] = *itop;\n }\n /* Update STP, etc. */\n npoin[*itop] = npoin[itmp];\n npoin[itmp] = *itop;\n stp\t [*itop] = *pastp;\n ifrq [*itop] = *ifreq;\n nl\t [*itop] = -1;\n nr\t [*itop] = -1;\n}\n\n\nRboolean f6xact(int nrow, int *irow, int *kyy, int *key, int *ldkey, int *last, int *ipn) {\n/* -----------------------------------------------------------------------\n Name:\t F6XACT aka \"GET\"\n Purpose: Pop a node off the stack.\n\n Arguments:\n NROW - The number of rows in the table.\t\t\t(Input)\n IROW - Vector of length nrow containing the row sums on\n output.\t\t\t\t\t\t(Output)\n KYY - Constant mutlipliers used in forming the hash\n table key.\t\t\t\t\t(Input)\n KEY - Vector of length LDKEY containing the hash table\n keys.\t\t\t\t\t\t(In/out)\n LDKEY - Length of vector KEY.\t\t\t\t(Input)\n LAST - Index of the last key popped off the stack.\t(In/out)\n IPN - Pointer to the linked list of past path lengths.\t(Output)\n\n Return value :\n TRUE if there are no additional nodes to process. (Output)\n ----------------------------------------------------------------------- */\n int kval, j;\n\n --key;\n\nL10:\n ++(*last);\n if (*last <= *ldkey) {\n\tif (key[*last] < 0)\n\t goto L10;\n\n\t/* Get KVAL from the stack */\n\tkval = key[*last];\n\tkey[*last] = -9999;\n\tfor (j = nrow-1; j > 0; j--) {\n\t irow[j] = kval / kyy[j];\n\t kval -= irow[j] * kyy[j];\n\t}\n\tirow[0] = kval;\n\t*ipn = *last;\n\treturn FALSE;\n } else {\n\t*last = 0;\n\treturn TRUE;\n }\n}\n\n\nvoid f7xact(int nrow, int *imax, int *idif, int *k, int *ks, int *iflag) {\n/* -----------------------------------------------------------------------\n Name:\t F7XACT\n Purpose: Generate the new nodes for given marginal totals.\n\n Arguments:\n NROW - The number of rows in the table.\t\t\t(Input)\n IMAX - The row marginal totals.\t\t\t\t(Input)\n IDIF - The column counts for the new column.\t\t(in/out)\n K\t - Indicator for the row to decrement.\t\t(in/out)\n KS\t - Indicator for the row to increment.\t\t(in/out)\n IFLAG - Status indicator.\t\t\t\t\t(Output)\n\t If IFLAG is zero, a new table was generated. For\n\t IFLAG = 1, no additional tables could be generated.\n ----------------------------------------------------------------------- */\n int i, m, kk, mm;\n\n /* Parameter adjustments */\n --idif;\n --imax;\n\n /* Function Body */\n *iflag = 0;\n /* Find node which can be incremented, ks */\n if (*ks == 0)\n\tdo {\n\t ++(*ks);\n\t} while (idif[*ks] == imax[*ks]);\n\n /* Find node to decrement (>ks) */\n if (idif[*k] > 0 && *k > *ks) {\n\t--idif[*k];\n\tdo {\n\t --(*k);\n\t} while(imax[*k] == 0);\n\n\tm = *k;\n\n\t/* Find node to increment (>=ks) */\n\twhile (idif[m] >= imax[m]) {\n\t --m;\n\t}\n\t++idif[m];\n\t/* Change ks */\n\tif (m == *ks && idif[m] == imax[m])\n\t *ks = *k;\n }\n else {\n Loop:\n\t/* Check for finish */\n\tfor (kk = *k + 1; kk <= nrow; ++kk) {\n\t if (idif[kk] > 0) {\n\t\tgoto L70;\n\t }\n\t}\n\t*iflag = 1;\n\treturn;\n\n L70:\n\t/* Reallocate counts */\n\tmm = 1;\n\tfor (i = 1; i <= *k; ++i) {\n\t mm += idif[i];\n\t idif[i] = 0;\n\t}\n\t*k = kk;\n\n\tdo {\n\t --(*k);\n\t m = imin2(mm, imax[*k]);\n\t idif[*k] = m;\n\t mm -= m;\n\t} while (mm > 0 && *k != 1);\n\n\t/* Check that all counts reallocated */\n\tif (mm > 0) {\n\t if (kk != nrow) {\n\t\t*k = kk;\n\t\tgoto Loop;\n\t }\n\t *iflag = 1;\n\t return;\n\t}\n\t/* Get ks */\n\t--idif[kk];\n\t*ks = 0;\n\tdo {\n\t ++(*ks);\n\t if (*ks > *k) {\n\t\treturn;\n\t }\n\t} while (idif[*ks] >= imax[*ks]);\n }\n}\n\n\nvoid f8xact(int *irow, int is, int i1, int izero, int *new) {\n/* -----------------------------------------------------------------------\n Name:\t F8XACT\n Purpose: Routine for reducing a vector when there is a zero\n\t element.\n Arguments:\n IROW - Vector containing the row counts.\t\t\t(Input)\n IS\t - Indicator.\t\t\t\t\t(Input)\n I1\t - Indicator.\t\t\t\t\t(Input)\n IZERO - Position of the zero.\t\t\t\t(Input)\n NEW - Vector of new row counts.\t\t\t\t(Output)\n ----------------------------------------------------------------------- */\n\n int i;\n\n /* Parameter adjustments */\n --new;\n --irow;\n\n /* Function Body */\n for (i = 1; i < i1; ++i)\n new[i] = irow[i];\n\n for (i = i1; i <= izero - 1; ++i) {\n if (is >= irow[i + 1]) break;\n new[i] = irow[i + 1];\n }\n\n new[i] = is;\n\n for(;;) {\n ++i;\n if (i > izero) return;\n new[i] = irow[i];\n }\n}\n\nstatic double f9xact(int n, int ntot, int *ir, double *fact) {\n/* -----------------------------------------------------------------------\n Name:\t F9XACT\n Purpose: Computes the log of a multinomial coefficient.\n\n Arguments:\n N\t - Length of IR.\t\t\t\t\t(Input)\n NTOT - Number for factorial in numerator.\t\t(Input)\n IR\t - Vector of length N containing the numbers for\n the denominator of the factorial.\t\t\t(Input)\n FACT - Table of log factorials.\t\t\t\t(Input)\n Returns:\n \t - The log of the multinomal coefficient.\t\t(Output)\n ----------------------------------------------------------------------- */\n double d = fact[ntot];\n for (int k = 0; k < n; k++)\n d -= fact[ir[k]];\n return d;\n}\n\n\nRboolean f10act(int nrow, int *irow, int ncol, int *icol, double *val,\n double *fact, int *nd, int *ne, int *m) {\n/* -----------------------------------------------------------------------\n Name:\t F10ACT\n Purpose: Computes the shortest path length for special tables.\n\n Arguments:\n NROW - The number of rows in the table.\t\t\t(Input)\n IROW - Vector of length NROW containing the row totals.\t(Input)\n NCOL - The number of columns in the table. \t\t(Input)\n ICO - Vector of length NCOL containing the column totals.(Input)\n VAL - The shortest path. \t\t\t\t(Input/Output)\n FACT - Vector containing the logarithms of factorials. (Input)\n ND\t - Workspace vector of length NROW.\t\t\t(Input)\n NE\t - Workspace vector of length NCOL.\t\t\t(Input)\n M\t - Workspace vector of length NCOL.\t\t\t(Input)\n\n Returns (VAL and):\n XMIN - Set to true if shortest path obtained. \t\t(Output)\n ----------------------------------------------------------------------- */\n int i, is, ix;\n\n for (i = 0; i < nrow - 1; ++i)\n nd[i] = 0;\n\n is = icol[0] / nrow;\n ix = icol[0] - nrow * is;\n ne[0] = is;\n m[0] = ix;\n if (ix != 0) ++nd[ix-1];\n\n for (i = 1; i < ncol; ++i) {\n ix = icol[i] / nrow;\n ne[i] = ix;\n is += ix;\n ix = icol[i] - nrow * ix;\n m[i] = ix;\n if (ix != 0) ++nd[ix-1];\n }\n\n for (i = nrow - 3; i >= 0; --i)\n nd[i] += nd[i + 1];\n\n ix = 0;\n for (i = nrow; i >= 2; --i) {\n ix += is + nd[nrow - i] - irow[i-1];\n if (ix < 0) return FALSE;\n }\n\n for (i = 0; i < ncol; ++i) {\n ix = ne[i];\n is = m[i];\n *val += is * fact[ix + 1] + (nrow - is) * fact[ix];\n }\n return TRUE;\n}\n\nvoid f11act(int *irow, int i1, int i2, int *new) {\n/* -----------------------------------------------------------------------\n Name:\t F11ACT\n Purpose: Routine for revising row totals.\n\n Arguments:\n IROW - Vector containing the row totals.\t(Input)\n I1\t - Indicator.\t\t\t(Input)\n I2\t - Indicator. \t\t\t(Input)\n NEW - Vector containing the row totals.\t(Output)\n ----------------------------------------------------------------------- */\n int i;\n for (i = 0; i < (i1 - 1); ++i)\tnew[i] = irow[i];\n for (i = i1; i <= i2; ++i)\t new[i-1] = irow[i];\n}\n\nstatic int iwork(int iwkmax, int *iwkpt, int number, int itype) {\n/* -----------------------------------------------------------------------\n Name:\t iwork\n Purpose: Routine for allocating workspace.\n\n Arguments:\n iwkmax - Maximum (int) amount of workspace.\t\t(Input)\n iwkpt - Amount of (int) workspace currently allocated.\t(in/out)\n number - Number of elements of workspace desired.\t\t(Input)\n itype - Workspace type.\t\t\t\t\t(Input)\n\t ITYPE TYPE\n\t\t2 integer\n\t\t3 float\n\t\t4 double\n iwork(): Index in rwrk, dwrk, or iwrk of the beginning of\n the first free element in the workspace array.\t(Output)\n ----------------------------------------------------------------------- */\n int i = *iwkpt;\n if (itype == 2 || itype == 3)\n *iwkpt += number;\n else { /* double */\n if (i % 2 != 0) ++i;\n *iwkpt += (number << 1);\n i /= 2;\n }\n Apop_stopif(*iwkpt >iwkmax, has_error=true;return i, 0, \"Out of workspace: %i > %i\", *iwkpt, iwkmax);\n return i;\n}\n\n#ifndef USING_R\n\nvoid isort(int *n, int *ix) {\n/* -----------------------------------------------------------------------\n Name:\t ISORT\n Purpose: Shell sort for an int vector.\n\n Arguments:\n N\t - Lenth of vector IX.\t(Input)\n IX\t - Vector to be sorted.\t(in/out)\n ----------------------------------------------------------------------- */\n static int ikey, i, j, m, il[10], kl, it, iu[10], ku;\n\n /* Parameter adjustments */\n --ix;\n\n /* Function Body */\n m = 1;\n i = 1;\n j = *n;\n\nL10:\n if (i >= j) \n goto L40;\n kl = i;\n ku = j;\n ikey = i;\n ++j;\n /* Find element in first half */\nL20:\n ++i;\n if (i < j) \n if (ix[ikey] > ix[i]) \n goto L20;\n /* Find element in second half */\nL30:\n --j;\n if (ix[j] > ix[ikey]) {\n goto L30;\n }\n /* Interchange */\n if (i < j) {\n it = ix[i];\n ix[i] = ix[j];\n ix[j] = it;\n goto L20;\n }\n it = ix[ikey];\n ix[ikey] = ix[j];\n ix[j] = it;\n /* Save upper and lower subscripts of the array yet to be sorted */\n if (m < 11) {\n if (j - kl < ku - j) {\n il[m - 1] = j + 1;\n iu[m - 1] = ku;\n i = kl;\n --j;\n } else {\n il[m - 1] = kl;\n iu[m - 1] = j - 1;\n i = j + 1;\n j = ku;\n }\n ++m;\n goto L10;\n } else Apop_stopif(1, return, 0, \"This should never occur.\");\n /* Use another segment */\nL40:\n --m;\n if (m == 0) \n return;\n i = il[m - 1];\n j = iu[m - 1];\n goto L10;\n}\n\nstatic double gammds(double *y, double *p, int *ifault) {\n/* -----------------------------------------------------------------------\n Name:\t GAMMDS\n Purpose: Cumulative distribution for the gamma distribution.\n Usage: PGAMMA (Q, ALPHA,IFAULT)\n Arguments:\n Q\t - Value at which the distribution is desired. (Input)\n ALPHA - Parameter in the gamma distribution. (Input)\n IFAULT - Error indicator.\t(Output)\n\t IFAULT DEFINITION\n\t\t 0 No error\n\t\t 1 An argument is misspecified.\n\t\t 2 A numerical error has occurred.\n PGAMMA - The cdf for the gamma distribution with parameter alpha\n\t evaluated at Q. (Output)\n -----------------------------------------------------------------------\n\n Algorithm AS 147 APPL. Statist. (1980) VOL. 29, P. 113\n\n Computes the incomplete gamma integral for positive parameters Y, P\n using and infinite series.\n */\n\n static double a, c, f, g;\n\n /* Checks for the admissibility of arguments and value of F */\n *ifault = 1;\n g = 0.;\n if (*y <= 0. || *p <= 0.)\n return g;\n *ifault = 2;\n\n /*\n ALOGAM is natural log of gamma function no need to test ifail as\n an error is impossible\n\n BK edit: using gsl_sf_lngamma instead. It has more methods--> maybe slower; more precise.\n\n */\n\n a = *p + 1.;\n f = exp(*p * log(*y) - gsl_sf_lngamma(a) - *y);\n if (f == 0.) \n return g;\n *ifault = 0;\n\n /* Series begins */\n c = 1.;\n g = 1.;\n a = *p;\n do {\n a += 1.;\n c *= (*y / a);\n g += c;\n } while (c > 1e-6 * g);\n\n g *= f;\n return g;\n}\n\n/** Convert from an \\ref apop_data set to a table of integers.\n\nNot too necessary, but I needed it for the Fisher exact test.\n*/\nstatic int *apop_data_to_int_array(apop_data *intab){\n int rowct = intab->matrix->size1,\n colct = intab->matrix->size2,\n *out = malloc(sizeof(int)*(rowct* colct));\n for (int i=0; i< rowct; i++)\n for (int j=0; j< colct; j++)\n out[j*rowct + i] = (int) gsl_matrix_get(intab->matrix, i, j);\n return out;\n}\n\n/** Run the Fisher exact test on an input contingency table.\n\n\\return An \\ref apop_data set with two rows:
\n \"probability of table\": Probability of the observed table for fixed marginal totals.\t
\n \"p value\": Table p-value.\tThe probability of a more extreme table,\n\t where `extreme' is in a probabilistic sense.\n\n\\li If there are processing errors, these values will be NaN.\n\n\\exception out->error=='p' Processing error in the test.\n\nFor example: \n\n\\include test_fisher.c\n*/\napop_data *apop_test_fisher_exact(apop_data *intab){\n double prt, pre,\n expect = -1,\n percent = 80,\n emin = 1;\n int *intified = apop_data_to_int_array(intab),\n workspace = 200000,\n mult = 30,\n rowct = intab->matrix->size1,\n colct = intab->matrix->size2;\n has_error=0;\nOMP_critical (fexact) //f3xact and f5exact use static vars for some state-keeping.\n fexact(&rowct, \n &colct,\n intified,\n &rowct,\n // Cochran condition for asym.chisq. decision:\n &expect,\n &percent,\n &emin,\n &prt,\n &pre,\n &workspace,\n &mult);\n free(intified);\n apop_data *out = apop_data_alloc();\n Asprintf(&out->names->title, \"Fisher Exact test\");\n apop_data_add_named_elmt(out, \"probability of table\", prt);\n apop_data_add_named_elmt(out, \"p value\", pre);\n Apop_stopif(has_error, out->error='p'; return out, 0, \"processing error; don't trust the results.\");\n return out;\n}\n#endif /* not USING_R */\n" }, { "path": "apop_hist.m4.c", "content": "/** \\file apop_hist.c */\n/* Functions that work with PMFs and histograms.\n\nCopyright (c) 2006--2007, 2010, 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. \n (Except psmirnov2x, Copyright R Project, but also licensed under the GPL.)\n*/\n\n#include \"apop_internal.h\"\n#include \n#include \n#include \n\n/** Make random draws from an \\ref apop_model, and bin them using a binspec in the style\n of \\ref apop_data_to_bins. If you have a data set that used the same binspec, you now have synced histograms, which you can plot or sensibly test hypotheses about.\n\n\n\\param binspec A description of the bins in which to place the draws; see \\ref apop_data_to_bins. (default: as in \\ref apop_data_to_bins.)\n\\param model The model to be drawn from. Because this function works via random draws, the model needs to have a \n\\c draw method. (No default)\n\\param draws The number of random draws to make. (arbitrary default = 10,000)\n\\param bin_count If no bin spec, the number of bins to use (default: as per \\ref apop_data_to_bins, \\f$\\sqrt(N)\\f$)\n\n\\return An \\ref apop_pmf model, with a new binned data set attached (which you may\nhave to apop_data_free(output_model->data) to prevent memory leaks). The\nweights on the data set are normalized to sum to one.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_model *apop_model_to_pmf(apop_model *model, apop_data *binspec, long int draws, int bin_count){\n apop_model* apop_varad_var(model, NULL);\n Apop_assert(model && model->draw, \"The second argument needs to be an apop_model with a 'draw' function \"\n \"that I can use to make random draws.\");\n apop_data* apop_varad_var(binspec, NULL);\n int apop_varad_var(bin_count, 0);\n long int apop_varad_var(draws, 1e4);\nAPOP_VAR_ENDHEAD\n Get_vmsizes(binspec);\n apop_data *outd = apop_model_draws(model, draws);\n apop_data *outbinned = apop_data_to_bins(outd, binspec, .bin_count=bin_count);\n apop_data_free(outd);\n apop_vector_normalize(outbinned->weights);\n return apop_estimate(outbinned, apop_pmf);\n} \n\n/** Test the goodness-of-fit between two \\ref apop_pmf models. \n\nLet \\f$o_i\\f$ be the \\f$i\\f$th observed bin and \\f$e_i\\f$ the expected value of that\nbin; then under typical assumptions, $\\f$\\Sum_i^N (o_i-e_i)^2/e_i \\sim \\Chi^2_{N-1}\\f$.\n\nIf you send two histograms, I assume that the histograms are synced: for PMFs,\nyou've used \\ref apop_data_to_bins to generate two histograms using the same binspec,\nor you've used \\ref apop_data_pmf_compress to guarantee that each observation value\nappears exactly once in each data set.\n\nIn any case, all values in the \\c observed set must appear in the \\c\nexpected set with nonzero weight; otherwise this will return a \\f$\\chi^2\\f$ statistic\nof \\c GSL_POSINF, indicating that it is impossible for the \\c observed data to have\nbeen drawn from the \\c expected distribution.\n\n\\li If an observation row has weight zero, I skip it. if apop_opts.verbose >=1 I will show a warning.\n*/\napop_data *apop_histograms_test_goodness_of_fit(apop_model *observed, apop_model *expected){\n int df = observed->data->weights->size;\n double diff = 0;\n for (int i=0; i< observed->data->weights->size; i++){\n double obs_val = gsl_vector_get(observed->data->weights, i);\n double exp_val = apop_p(Apop_r(observed->data, i), expected);\n if (exp_val == 0){\n diff = GSL_POSINF; \n break;\n }\n if (obs_val==0){\n Apop_notify(1, \"element %i of the observed data has weight zero. Skipping it.\", i);\n df --;\n } else \n diff += gsl_pow_2(obs_val - exp_val)/exp_val;\n }\n //Data gathered. Now output\n apop_data *out = apop_data_alloc();\n double toptail = gsl_cdf_chisq_Q(diff, df-1);\n Asprintf(&out->names->title, \"Goodness-of-fit test via Chi-squared statistic\");\n apop_data_add_named_elmt(out, \"Chi squared statistic\", diff);\n apop_data_add_named_elmt(out, \"df\", df-1);\n apop_data_add_named_elmt(out, \"p value\", toptail); \n apop_data_add_named_elmt(out, \"confidence\", 1 - toptail);\n return out;\n}\n\n/*Everything from here to psmirnov2x (inclusive) is cut/pasted/trivially modified from the R project. Copyright them. */\nstatic void m_multiply(long double *A, long double *B, long double *C, int m) {\n /* Auxiliary routine used by K().\n Matrix multiplication.\n */\n for (int i = 0; i < m; i++)\n for (int j = 0; j < m; j++) {\n long double s = 0;\n for (int k = 0; k < m; k++)\n s += A[i * m + k] * B[k * m + j];\n C[i * m + j] = s;\n }\n}\n\nstatic void m_power(long double *A, int eA, long double *V, int *eV, int m, int n) {\n /* Auxiliary routine used by K().\n Matrix power.\n */\n long double *B;\n int eB, i;\n\n if (n == 1) {\n for (i = 0; i < m * m; i++)\n V[i] = A[i];\n *eV = eA;\n return;\n }\n m_power(A, eA, V, eV, m, n / 2);\n B = calloc(m * m, sizeof(long double));\n m_multiply(V, V, B, m);\n eB = 2 * (*eV);\n if ((n % 2) == 0) {\n for (i = 0; i < m * m; i++)\n V[i] = B[i];\n *eV = eB;\n } else {\n m_multiply(A, B, V, m);\n *eV = eA + eB;\n }\n if (V[(m / 2) * m + (m / 2)] > 1e140) {\n for (i = 0; i < m * m; i++)\n V[i] = V[i] * 1e-140;\n *eV += 140;\n }\n free(B);\n}\n\n/* The two-sided one-sample 'exact' distribution */\nstatic double kolmogorov_2x(int n, double d) {\n /* Compute Kolmogorov's distribution.\n Code published in\n\t George Marsaglia and Wai Wan Tsang and Jingbo Wang (2003),\n\t \"Evaluating Kolmogorov's distribution\".\n\t Journal of Statistical Software, Volume 8, 2003, Issue 18.\n\t URL: http://www.jstatsoft.org/v08/i18/.\n */\n\n int k, m, i, j, g, eH, eQ;\n long double h, s, *H, *Q;\n\n /* The faster right-tail approximation is omitted here.\n s = d*d*n; \n if(s > 7.24 || (s > 3.76 && n > 99)) \n return 1-2*exp(-(2.000071+.331/sqrt(n)+1.409/n)*s);\n */\n k = (n * d) + 1;\n m = 2 * k - 1;\n h = k - n * d;\n H = calloc(m * m, sizeof(long double));\n Q = calloc(m * m, sizeof(long double));\n for(i = 0; i < m; i++)\n for(j = 0; j < m; j++)\n if (i - j + 1 < 0) H[i * m + j] = 0;\n else H[i * m + j] = 1;\n for(i = 0; i < m; i++) {\n H[i * m] -= pow(h, i + 1);\n H[(m - 1) * m + i] -= pow(h, (m - i));\n }\n H[(m - 1) * m] += ((2 * h - 1 > 0) ? pow(2 * h - 1, m) : 0);\n for(i = 0; i < m; i++)\n for(j=0; j < m; j++)\n if(i - j + 1 > 0)\n for(g = 1; g <= i - j + 1; g++)\n H[i * m + j] /= g;\n eH = 0;\n m_power(H, eH, Q, &eQ, m, n);\n s = Q[(k - 1) * m + k - 1];\n for(i = 1; i <= n; i++) {\n s = s * i / n;\n if(s < 1e-140) {\n s *= 1e140;\n eQ -= 140;\n }\n }\n s *= pow(10., eQ);\n free(H);\n free(Q);\n return(s);\n}\n\nstatic double psmirnov2x(double x, int m, int n) {\n if(m > n) {\n int tmp = n; n = m; m = tmp;\n }\n double md = m;\n double nd = n;\n // q has 0.5/mn added to ensure that rounding error doesn't\n // turn an equality into an inequality, eg abs(1/2-4/5)>3/10\n long double q = (.5+floor(x * md * nd - 1e-7)) / (md * nd);\n long double u[n+1];\n\n for(int j = 0; j <= n; j++) \n u[j] = ((j / nd) > q) ? 0 : 1;\n for(int i = 1; i <= m; i++) {\n long double w = i/(i + nd);\n u[0] = (i / md) > q \n ? 0\n : w * u[0];\n for(int j = 1; j <= n; j++) \n u[j] = fabs(i / md - j / nd) > q\n ? 0\n : w * u[j] + u[j - 1];\n }\n return u[n];\n}\n\n/** Run the Kolmogorov-Smirnov test to determine whether two distributions are identical.\n\n\\param m1 A sorted PMF model. I.e., a model estimated via something like \napop_model *m1 = apop_estimate(apop_data_sort(input_data), apop_pmf);\n\n\\param m2 Another \\ref apop_model. If it is a PMF, then I will use a two-sample test,\nwhich is different from the one-sample test used if this is not a PMF.\n\n\\return An \\ref apop_data set including the \\f$p\\f$-value from the Kolmogorov-Smirnov\ntest that the two distributions are equal.\n\n\\exception out->error='m' Model error: \\c m1 is not an \\ref apop_pmf. I verify this\nby checking whether m1->cdf == apop_pmf->cdf.\n\n\\li If you are using a \\ref apop_pmf model, the data set(s) must be sorted before\nyou set up the model, as per the example below. See \\ref apop_data_sort and the\ndiscussion of CDFs in the \\ref apop_pmf documentation. If you don't do this, the test\nwill almost certainly reject the null hypothesis that \\c m1 and \\c m2 are identical.\nA future version of Apophenia may implement a mechanism to allow this function to test\nfor sorted data, but it currently can't.\n\nHere is an example, which tests whether a set of draws from a Normal(0, 1) matches a\nsequence of Normal distributions with increasing mean.\n\n\\include ks_tests.c\n*/\napop_data *apop_test_kolmogorov(apop_model *m1, apop_model *m2){\n Apop_stopif(m1->cdf != apop_pmf->cdf, apop_return_data_error('m'), \n 0, \"First model has to be a PMF. I check whether m1->cdf == apop_pmf->cdf.\");\n bool m2_is_pmf = (m2->cdf == apop_pmf->cdf);\n\n int maxsize1, maxsize2;\n {Get_vmsizes(m1->data); maxsize1 = maxsize;} //copy one of the macro's variables \n {Get_vmsizes(m2->data); maxsize2 = maxsize;} //to the full function's scope.\n double largest_diff = GSL_NEGINF;\n double sum = 0;\n for (size_t i=0; i< maxsize1; i++){\n apop_data *arow = Apop_r(m1->data, i);\n sum += m1->data->weights ? gsl_vector_get(m1->data->weights, i) : 1./maxsize1;\n largest_diff = GSL_MAX(largest_diff, fabs(sum-apop_cdf(arow, m2)));\n }\n if (m2_is_pmf){\n double sum = 0;\n for (size_t i=0; i< maxsize2; i++){ //There could be matched data rows to m1, so there is redundancy.\n apop_data *arow = Apop_r(m2->data, i); // Feel free to submit a smarter version.\n sum += m2->data->weights ? gsl_vector_get(m2->data->weights, i) : 1./maxsize2;\n largest_diff = GSL_MAX(largest_diff, fabs(sum-apop_cdf(arow, m2)));\n }\n }\n apop_data *out = apop_data_alloc();\n Asprintf(&out->names->title, \"Kolmogorov-Smirnov test\");\n apop_data_add_named_elmt(out, \"max distance\", largest_diff);\n double ps = m2_is_pmf ? psmirnov2x(largest_diff, maxsize1, maxsize2)\n : kolmogorov_2x(maxsize1, largest_diff);\n apop_data_add_named_elmt(out, \"p value, 2 tail\", 1-ps);\n apop_data_add_named_elmt(out, \"confidence, 2 tail\", ps);\n return out;\n}\n\n/** Create a histogram from data by putting data into bins of fixed width. Your input\n\\ref apop_data set may be multidimensional, and may include both vector and matrix\nparts, and the bins output will have corresponding dimension.\n\n\\param indata The input data that will be binned, one observation per row. This is\n copied and the copy will be modified. (No default)\n\\param binspec This is an \\ref apop_data set with the same number of columns as \\c indata. \n If you want a fixed size for the bins, then the first row of the bin spec is the\n bin width for each column. This allows you to specify a width for each dimension,\n or specify the same size for all with something like:\n\\code\napop_data *binspec = apop_data_copy(Apop_r(indata, 0));\ngsl_matrix_set_all(binspec->matrix, 10); //bins of size 10 for all dim.s\napop_data_to_bins(indata, binspec);\n\\endcode\n The presumption is that the first bin starts at zero in all cases. You can add a second\n row to the spec to give the offset for each dimension. (default: NULL)\n\\param bin_count If you don't provide a bin spec, I'll provide this many evenly-sized bins to cover the data set. (Default: \\f$\\sqrt{N}\\f$)\n\\param close_top_bin Normally, a bin covers the range from the point equal to its\n minimum to points strictly less than the minimum plus the width. if \\c 'y', then\n the top bin includes points less than or equal to the upper bound. This solves the\n problem of displaying histograms where the top bin is just one point. (default:\n \\c 'y' if \\c binspec==NULL, else \\c 'n')\n\n\\return A pointer to an \\ref apop_data set with the same dimension as your input data.\nEach cell is an integer giving the bin number into which the cell falls.\n\n\\li If no binspec and no binlist, then a grid with offset equal to the min of the\n column, and bin size such that it takes \\f$\\sqrt{N}\\f$ bins to cover the range to the\n max element.\n\\li The text segment is not binned. The \\c more pointer, if any, is not followed.\n\\li If there are weights, they are copied to the output via \\ref apop_vector_copy.\n\\li Given \\c NULL input, return \\c NULL output. Print a warning if apop_opts.verbose >= 2.\n\nIff you didn't give me a binspec, then I attach one to the output set as a page named\n\\c \\. This means that you can snap a second data set to the same grid using\n\\code\napop_data_to_bins(first_set, NULL);\napop_data_to_bins(second_set, apop_data_get_page(first_set, \"\"));\n\\endcode\n\\li The output has exactly as many rows as the input. Because many rows will be identical\nafter binning, it may be fruitful to run it through \\ref apop_data_pmf_compress to\nproduce a short list with one total weight per bin.\n\nHere is a sample program highlighting \\ref apop_data_to_bins and \\ref apop_data_pmf_compress .\n\n\\include binning.c\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_data_to_bins(apop_data const *indata, apop_data const *binspec, int bin_count, char close_top_bin){\n apop_data const *apop_varad_var(indata, NULL);\n Apop_assert_c(indata, NULL, 2, \"NULL input data set, so returning NULL output data set.\");\n apop_data const *apop_varad_var(binspec, NULL);\n char apop_varad_var(close_top_bin, binspec==NULL ? 'y' : 'n');\n int apop_varad_var(bin_count, 0);\nAPOP_VAR_ENDHEAD\n Get_vmsizes(indata); //firstcol, vsize, msize1, msize2\n double binwidth, offset, max=0;\n apop_data *out = apop_data_alloc(vsize, msize1, msize2);\n apop_data const *bs = binspec ? binspec\n : apop_data_add_page(out, \n apop_data_alloc(vsize? 2: 0, msize1? 2: 0, indata->matrix ? msize2: 0),\n \"\");\n for (int j= firstcol; j< msize2; j++){\n gsl_vector *onecol = Apop_cv(out, j);\n gsl_vector *datacol = Apop_cv(indata, j);\n if (binspec){\n binwidth = apop_data_get(binspec, 0, j);\n offset = ((binspec->vector && binspec->vector->size==2 )\n ||(binspec->matrix && binspec->matrix->size1==2)) ? apop_data_get(binspec, 1, j) : 0;\n } else {\n gsl_vector *abin = Apop_cv(bs, j);\n max = gsl_vector_max(datacol);\n gsl_vector_set(abin, 1, offset = gsl_vector_min(datacol));\n gsl_vector_set(abin, 0, binwidth = (max - offset)/(bin_count ? bin_count : sqrt(datacol->size)));\n }\n for (int i=0; i< onecol->size; i++){\n double val = gsl_vector_get(datacol, i);\n double adjust = (close_top_bin=='y' && val == max && val!=offset) ? 2*GSL_DBL_EPSILON : 0;\n gsl_vector_set(onecol, i, (floor((val-offset-adjust)/binwidth))*binwidth+offset);\n }\n }\n if (indata->weights) out->weights = apop_vector_copy(indata->weights);\n return out;\n}\n\n/** Return a new vector that is the moving average of the input vector.\n\n\\param v The input vector, unsmoothed\n\\param bandwidth An integer \\f$\\geq 1\\f$ giving the number of elements to be averaged to produce one number.\n\\return A smoothed vector of size v->size - (bandwidth/2)*2.\n */\ngsl_vector *apop_vector_moving_average(gsl_vector *v, size_t bandwidth){\n Apop_stopif(!v, return NULL, 0, \"You asked me to smooth a NULL vector; returning NULL.\");\n Apop_stopif(!bandwidth, return apop_vector_copy(v), 0, \"Bandwidth must be >=1. Returning a copy of original vector with no smoothing.\");\n int halfspan = bandwidth/2;\n Apop_stopif((v->size - halfspan*2)<=0, return NULL, 0, \"Bandwidth wider than the vector. Returning NULL.\");\n gsl_vector *vout = gsl_vector_calloc(v->size - halfspan*2);\n for(size_t i=0; i < vout->size; i ++){\n double *item = gsl_vector_ptr(vout, i);\n for (int j=-halfspan; j < halfspan+1; j ++)\n *item += gsl_vector_get(v, j+ i+ halfspan);\n *item /= halfspan*2 +1;\n }\n return vout;\n}\n" }, { "path": "apop_internal.h", "content": "/* These are functions used here and there to write Apophenia. They're\n not incredibly useful, or even very good form, so they're not public. Cut\n & paste `em into your own code if you'd like.\n */\n\n/* Many Apop functions try to treat the vector and matrix equally, which\n requires knowing which exists and what the sizes are. */\n#define Get_vmsizes(d) \\\n int firstcol = d && (d)->vector ? -1 : 0; \\\n int vsize = d && (d)->vector ? (d)->vector->size : 0; \\\n int wsize = d && (d)->weights ? (d)->weights->size : 0; \\\n int msize1 = d && (d)->matrix ? (d)->matrix->size1 : 0; \\\n int msize2 = d && (d)->matrix ? (d)->matrix->size2 : 0; \\\n int tsize = vsize + msize1*msize2; \\\n int maxsize = GSL_MAX(vsize, GSL_MAX(msize1, d?d->textsize[0]:0));\\\n (void)(tsize||wsize||firstcol||maxsize) /*prevent unused variable complaints */;\n\n// Define a static variable, and initialize on first use.\n#define Staticdef(type, name, def) static type (name) = NULL; if (!(name)) (name) = (def);\n\n// Check for NULL and complain if so.\n#define Nullcheck(in, errval) Apop_assert_c(in, errval, apop_errorlevel, \"%s is NULL.\", #in);\n#define Nullcheck_m(in, errval) Apop_assert_c(in, errval, apop_errorlevel, \"%s is a NULL model.\", #in);\n#define Nullcheck_mp(in, errval) Nullcheck_m(in, errval); Apop_assert_c((in)->parameters, errval, apop_errorlevel, \"%s is a model with NULL parameters. Please set the parameters and try again.\", #in);\n#define Nullcheck_d(in, errval) Apop_assert_c(in, errval, apop_errorlevel, \"%s is a NULL data set.\", #in);\n//And because I do them all so often:\n#define Nullcheck_mpd(data, model, errval) Nullcheck_m(model, errval); Nullcheck_p(model, errval); Nullcheck_d(data, errval);\n//deprecated:\n#define Nullcheck_p(in, errval) Nullcheck_mp(in, errval);\n\n//in apop_conversions.c Extend a string.\nvoid xprintf(char **q, char *format, ...);\n#define XN(in) ((in) ? (in) : \"\")\n\n//For a pedantic compiler. Continues on error, because there's not much else to do: the computer is clearly broken.\n#define Asprintf(...) Apop_stopif(asprintf(__VA_ARGS__)==-1, , 0, \"Error printing to a string.\")\n\n#include \n#include \nint apop_use_sqlite_prepared_statements(size_t col_ct);\nint apop_prepare_prepared_statements(char const *tabname, size_t col_ct, sqlite3_stmt **statement);\nchar *prep_string_for_sqlite(int prepped_statements, char const *astring);//apop_conversions.c\nvoid apop_gsl_error(char const *reason, char const *file, int line, int gsl_errno); //apop_linear_algebra.c\n\n//For when we're forced to use a global variable.\n#undef threadlocal\n#if __STDC_VERSION__ > 201100L\n #define threadlocal _Thread_local\n#elif defined(__APPLE__) \n #define threadlocal\n#elif defined(__GNUC__) && !defined(threadlocal)\n #define threadlocal __thread\n#else\n #define threadlocal\n#endif\n\n#ifdef _OPENMP\n#define PRAGMA(x) _Pragma(#x)\n#define OMP_critical(tag) PRAGMA(omp critical ( tag ))\n#define OMP_for(...) _Pragma(\"omp parallel for\") for(__VA_ARGS__)\n#define OMP_for_reduce(red, ...) PRAGMA(omp parallel for reduction( red )) for(__VA_ARGS__)\n#else\n#define OMP_critical(tag)\n#define OMP_for(...) for(__VA_ARGS__)\n#define OMP_for_reduce(red, ...) for(__VA_ARGS__)\n#endif\n\n#include \"config.h\"\n#ifndef HAVE___ATTRIBUTE__\n#define __attribute__(...)\n#endif\n\n#ifndef HAVE_ASPRINTF\n#include \n\n//asprintf, vararg, &c\nextern int asprintf (char **res, const char *format, ...)\n __attribute__ ((__format__ (__printf__, 2, 3)));\nextern int vasprintf (char **res, const char *format, va_list args)\n __attribute__ ((__format__ (__printf__, 2, 0)));\n#endif\n\n#include \"apop.h\"\nvoid add_info_criteria(apop_data *d, apop_model *m, apop_model *est, double ll, int param_ct); //In apop_mle.c\n\napop_model *maybe_prep(apop_data *d, apop_model *m, _Bool *is_a_copy); //in apop_mcmc, for apop_update.\n" }, { "path": "apop_linear_algebra.m4.c", "content": "/** \\file apop_linear_algebra.c\tAssorted things to do with matrices,\nsuch as take determinants or do singular value decompositions. Includes\nmany convenience functions that don't actually do math but add/delete\ncolumns, check bounds, et cetera.\n*/ \n/* Copyright (c) 2006--2007, 2012 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n\nvoid apop_gsl_error(const char *reason, const char *file, int line, int gsl_errno){\n Apop_notify(1, \"%s: %s\", file, reason);\n Apop_maybe_abort(1);\n}\n#define Checkgsl(...) if (__VA_ARGS__) {goto done;}\n#define Check_gsl_with_out(...) if (__VA_ARGS__) {out->error='m'; goto done;}\n#define Check_gsl_with_outmp(...) if (__VA_ARGS__) {gsl_matrix_free(*out); *out=NULL; goto done;}\n#define Set_gsl_handler gsl_error_handler_t *prior_handler = gsl_set_error_handler(apop_gsl_error);\n#define Unset_gsl_handler gsl_set_error_handler(prior_handler);\n\n/**\nCalculate the determinant of a matrix, its inverse, or both, via LU decomposition. The \\c in matrix is not destroyed in the process.\n\n\\see apop_matrix_determinant, apop_matrix_inverse\n\n\\param in The matrix to be inverted/determined. \n\\param out If you want an inverse, this is where to place the matrix to be filled with the inverse. Will be allocated by the function. \n\n\\param calc_det \n0: Do not calculate the determinant.
\n1: Do.\n\n\\param calc_inv\n0: Do not calculate the inverse.
\n1: Do.\n\n\\return If calc_det == 1, then return the determinant. Otherwise, just returns zero. If calc_inv!=0, \nthen \\c *out is pointed to the matrix inverse. In case of difficulty, I will set *out=NULL and return \\c NaN.\n*/\n\ndouble apop_det_and_inv(const gsl_matrix *in, gsl_matrix **out, int calc_det, int calc_inv) {\n Set_gsl_handler\n Apop_stopif(in->size1 != in->size2, *out=NULL; return GSL_NAN, 0, \"You asked me to invert a %zu X %zu matrix, \"\n \"but inversion requires a square matrix.\", in->size1, in->size2);\n int sign;\n double the_determinant = GSL_NAN;\n\tgsl_matrix *invert_me = gsl_matrix_alloc(in->size1, in->size1);\n\tgsl_permutation * perm = gsl_permutation_alloc(in->size1);\n\tgsl_matrix_memcpy (invert_me, in);\n\tCheckgsl(gsl_linalg_LU_decomp(invert_me, perm, &sign))\n\tif (calc_inv){\n\t\t*out = gsl_matrix_alloc(in->size1, in->size1); //square.\n\t\tCheck_gsl_with_outmp(gsl_linalg_LU_invert(invert_me, perm, *out))\n }\n\tif (calc_det)\n\t\tthe_determinant\t= gsl_linalg_LU_det(invert_me, sign);\n done:\n\tgsl_matrix_free(invert_me);\n\tgsl_permutation_free(perm);\n Unset_gsl_handler\n\treturn the_determinant;\n}\n\n/**\nInverts a matrix. The \\c in matrix is not destroyed in the process.\nYou may want to call \\ref apop_matrix_determinant first to check that your input is invertible, or use \\ref apop_det_and_inv to do both at once.\n\n\\param in The matrix to be inverted.\n\\return Its inverse.\n*/\ngsl_matrix * apop_matrix_inverse(const gsl_matrix *in) {\n gsl_matrix *out = NULL;\n apop_det_and_inv(in, &out, 0, 1);\n return out;\n}\n\n/**\nFind the determinant of a matrix. The \\c in matrix is not destroyed in the process.\n\nSee also \\ref apop_matrix_inverse , or \\ref apop_det_and_inv to do both at once.\n\n\\param in The matrix to be determined.\n\\return The determinant.\n*/\ndouble apop_matrix_determinant(const gsl_matrix *in) {\n return apop_det_and_inv(in, NULL, 1, 0);\n}\n\n/** Principal component analysis: hand in a matrix and (optionally) a number of desired dimensions, and I'll return a data set where each column of the matrix is an eigenvector. The columns are sorted, so column zero has the greatest weight. The vector element of the data set gives the weights.\n\nYou may also specify the number of elements your principal component space should have. If\nthis is equal to the rank of the space in which the input data lives, then the sum of\nweights will be one. If the dimensions desired is less than that (probably so you can\nprepare a plot), then the weights will be accordingly smaller, giving you an indication\nof how much variation these dimensions explain.\n\n\\param data The input matrix. I modify int in place so that each column has\nmean zero. (No default. If \\c NULL, return \\c NULL and print a warning iff\napop_opts.verbose >= 1.)\n\n\\param dimensions_we_want The singular value decomposition will return this many of the eigenvectors with the largest eigenvalues. (default: the size of the covariance matrix, i.e. data->size2)\n\n\\return Returns an \\ref apop_data set whose matrix is the principal component\nspace. Each column of the returned matrix will be another eigenvector; the columns\nwill be ordered by the eigenvalues.\n\nThe data set's vector will be the largest eigenvalues, scaled by the total of all eigenvalues (including those that were thrown out). The sum of these returned values will give you the percentage of variance explained by the factor analysis.\n\n\\exception out->error=='a' Allocation error.\n*/\nAPOP_VAR_HEAD apop_data * apop_matrix_pca(gsl_matrix *data, int const dimensions_we_want) {\n gsl_matrix * apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, 1, \"NULL data input\");\n int const apop_varad_var(dimensions_we_want, data->size2);\nAPOP_VAR_ENDHEAD\n Set_gsl_handler\n apop_data *pc_space\t= apop_data_alloc(0, data->size2, dimensions_we_want);\n Apop_stopif(pc_space->error, return pc_space, 0, \"Allocation error.\");\n\tpc_space->vector = gsl_vector_alloc(dimensions_we_want);\n Apop_stopif(!pc_space->vector, pc_space->error='a'; return pc_space, \n 0, \"Allocation error setting up a %i vector.\", dimensions_we_want);\n gsl_matrix *eigenvectors = gsl_matrix_alloc(data->size2, data->size2);\n gsl_vector *dummy_v \t = gsl_vector_alloc(data->size2);\n gsl_vector *all_evalues = gsl_vector_alloc(data->size2);\n gsl_matrix *square \t = gsl_matrix_calloc(data->size2, data->size2);\n Apop_stopif(!eigenvectors || !dummy_v || !all_evalues || !square, pc_space->error='a'; return pc_space, \n 0, \"Allocation error setting up workspace for %zu dimensions.\", data->size2);\n double eigentotals\t= 0;\n for (int i=0; i< data->size2; i++)\n apop_vector_normalize(Apop_mcv(data, i), NULL, 'm');\n\n\tCheckgsl(gsl_blas_dgemm(CblasTrans,CblasNoTrans, 1, data, data, 0, square))\n\tCheckgsl(gsl_linalg_SV_decomp(square, eigenvectors, all_evalues, dummy_v))\n\tfor (int i=0; i< all_evalues->size; i++)\n\t\teigentotals\t+= gsl_vector_get(all_evalues, i);\n\tfor (int i=0; imatrix, i, v);\n\t\tgsl_vector_set(pc_space->vector, i, gsl_vector_get(all_evalues, i)/eigentotals);\n\t}\n done:\n\tgsl_vector_free(dummy_v); \tgsl_vector_free(all_evalues);\n\tgsl_matrix_free(square); \tgsl_matrix_free(eigenvectors);\n Unset_gsl_handler\n return pc_space;\n}\n\nstatic void l10(double *d){ *d = log10(*d); }\nstatic void ln(double *d){ *d = log(*d); }\nstatic void ex(double *d){ *d = exp(*d); }\n\n/** Replace every vector element \\f$v_i\\f$ with log\\f$_{10}(v_i)\\f$.\n\\li If the input vector is \\c NULL, do nothing. \n*/\nvoid apop_vector_log10(gsl_vector *v){\n if (!v) return;\n apop_vector_apply(v, l10);\n}\n\n/** Replace every vector element \\f$v_i\\f$ with ln\\f$(v_i)\\f$.\n\\li If the input vector is \\c NULL, do nothing. \n*/\nvoid apop_vector_log(gsl_vector *v){\n if (!v) return;\n apop_vector_apply(v, ln);\n}\n\n/** Replace every vector element \\f$v_i\\f$ with exp\\f$(v_i)\\f$.\n\\li If the input vector is \\c NULL, do nothing. \n*/\nvoid apop_vector_exp(gsl_vector *v){\n if (!v) return;\n apop_vector_apply(v, ex);\n}\n\n/** Put the first vector on top of the second vector.\n\n\\param v1 the upper vector (default=\\c NULL, in which case this copies \\c v2)\n\\param v2 the second vector (default=\\c NULL, in which case nothing is added)\n\\param inplace If \\c 'y', use \\ref apop_vector_realloc to modify \\c v1 in place;\n see the caveats on that function. Otherwise, allocate a new vector, leaving \\c v1\n undisturbed. (default=\\c 'n')\n\\return the stacked data, either in a new vector or a pointer to \\c v1.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD gsl_vector *apop_vector_stack(gsl_vector *v1, gsl_vector const * v2, char inplace){\n gsl_vector * apop_varad_var(v1, NULL);\n gsl_vector const * apop_varad_var(v2, NULL);\n char apop_varad_var(inplace, 'n');\nAPOP_VAR_ENDHEAD\n gsl_vector *out;\n gsl_vector t;\n if (!v1 && v2){\n out = gsl_vector_alloc(v2->size);\n gsl_vector_memcpy(out, v2);\n return out;\n } else if (!v2 && v1){\n if (inplace == 'y')\n return v1;\n out = gsl_vector_alloc(v1->size);\n gsl_vector_memcpy(out, v1);\n return out;\n } else if (!v1 && !v2)\n return NULL;\n //else:\n size_t v1size = v1->size; //save in case of reallocing.\n if (inplace == 'y' )\n out = apop_vector_realloc(v1, v1->size+v2->size);\n else {\n out = gsl_vector_alloc(v1->size + v2->size);\n t = gsl_vector_subvector(out, 0, v1size).vector;\n gsl_vector_memcpy(&t, v1);\n }\n t = gsl_vector_subvector(out, v1size, v2->size).vector;\n gsl_vector_memcpy(&t, v2);\n return out;\n}\n\n/** Put the first matrix either on top of or to the right of the second matrix.\nReturns a new matrix, meaning that at the end of this function, until you \\c gsl_matrix_free() the original matrices, you will be taking up twice as much memory. Plan accordingly.\n\n\\param m1 the upper/rightmost matrix (default: \\c NULL, in which case this copies \\c m2)\n\\param m2 the second matrix (default: \\c NULL, in which case \\c m1 is returned)\n\\param posn If \\c 'r', stack rows on top of other rows. If \\c 'c' stack columns next to columns. (default: \\c 'r')\n\\param inplace If \\c 'y', use \\ref apop_matrix_realloc to modify \\c m1 in place; see the caveats on that function. Otherwise, allocate a new matrix, leaving \\c m1 undisturbed. (default: \\c 'n')\n\\return the stacked data, either in a new matrix or a pointer to \\c m1.\n\nFor example, here is a function to merge four matrices into a single two-part-by-two-part matrix. The original matrices are unchanged.\n\\code\ngsl_matrix *apop_stack_two_by_two(gsl_matrix *ul, gsl_matrix *ur, gsl_matrix *dl, gsl_matrix *dr){\n gsl_matrix *output, *t;\n output = apop_matrix_stack(ul, ur, 'c');\n t = apop_matrix_stack(dl, dr, 'c');\n apop_matrix_stack(output, t, 'r', .inplace='y');\n gsl_matrix_free(t);\n return output;\n}\n\\endcode\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD gsl_matrix *apop_matrix_stack(gsl_matrix *m1, gsl_matrix const * m2, char posn, char inplace){\n gsl_matrix *apop_varad_var(m1, NULL);\n gsl_matrix const *apop_varad_var(m2, NULL);\n char apop_varad_var(posn, 'r');\n char apop_varad_var(inplace, 'n');\nAPOP_VAR_ENDHEAD\n gsl_matrix *out;\n gsl_vector_view tmp_vector;\n if (!m1 && m2){\n out = gsl_matrix_alloc(m2->size1, m2->size2);\n gsl_matrix_memcpy(out, m2);\n return out;\n } else if (!m2 && m1) {\n if (inplace =='y')\n return m1;\n out = gsl_matrix_alloc(m1->size1, m1->size2);\n gsl_matrix_memcpy(out, m1);\n return out;\n } else if (!m2 && !m1) \n return NULL;\n\n if (posn == 'r'){\n Apop_stopif(m1->size2 != m2->size2, return NULL, 0, \"When stacking matrices on top of each other, they have to have the same number of columns, but m1->size2==%zu and m2->size2==%zu. Returning NULL.\", m1->size2, m2->size2);\n int m1size = m1->size1;\n if (inplace =='y')\n out = apop_matrix_realloc(m1, m1->size1 + m2->size1, m1->size2);\n else {\n out = gsl_matrix_alloc(m1->size1 + m2->size1, m1->size2);\n for (int i=0; i< m1size; i++){\n tmp_vector = gsl_matrix_row(m1, i);\n gsl_matrix_set_row(out, i, &(tmp_vector.vector));\n }\n }\n for (int i=m1size; i< m1size + m2->size1; i++){\n gsl_vector_const_view tmp_vector = gsl_matrix_const_row(m2, i- m1size);\n gsl_matrix_set_row(out, i, &(tmp_vector.vector));\n }\n return out;\n } else {\n Apop_stopif(m1->size1 != m2->size1, return NULL, 0, \"When stacking matrices side by side, \"\n \"they have to have the same number of rows, but m1->size1==%zu and m2->size1==%zu. Returning NULL.\"\n , m1->size1, m2->size1);\n int m1size = m1->size2;\n if (inplace =='y')\n out = apop_matrix_realloc(m1, m1->size1, m1->size2 + m2->size2);\n else {\n out = gsl_matrix_alloc(m1->size1, m1->size2 + m2->size2);\n for (int i=0; i< m1size; i++)\n gsl_matrix_set_col(out, i, Apop_mcv(m1, i));\n }\n for (int i=0; i< m2->size2; i++)\n gsl_matrix_set_col(out, i+ m1size, Apop_mcv((gsl_matrix*)m2, i));\n return out;\n } \n}\n\n/** Test that all elements of a vector are within bounds, so you can preempt a procedure\nthat is about to break on infinite or too-large values.\n\n\\param in A gsl_vector\n\\param max An upper and lower bound to the elements of the vector. (default: INFINITY)\n\\return 1 if everything is bounded: not Inf, -Inf, or NaN, and \\f$-\\max < x < \\max\\f$;
0 otherwise. \n \n\\li A \\c NULL vector has no unbounded elements, so \\c NULL input returns 1. You get a warning if apop_opts.verbosity >=2.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_vector_bounded(const gsl_vector *in, long double max){\n const gsl_vector * apop_varad_var(in, NULL)\n Apop_stopif(!in, return 1, 2, \"You sent in a NULL vector; returning 1.\");\n long double apop_varad_var(max, INFINITY)\nAPOP_VAR_END_HEAD\n for (size_t i=0; i< in->size; i++){\n double x = gsl_vector_get(in, i);\n if (!gsl_finite(x) || x> max || x< -max)\n return 0;\n }\n return 1;\n}\n\n\nstatic gsl_vector* dot_for_apop_dot(const gsl_matrix *m, const gsl_vector *v, \n const CBLAS_TRANSPOSE_t flip){\n #define Check_gslv(...) if (__VA_ARGS__) {gsl_vector_free(out); out=NULL;}\n gsl_vector *out = (flip ==CblasNoTrans)\n ? gsl_vector_calloc(m->size1)\n : gsl_vector_calloc(m->size2);\n Check_gslv(gsl_blas_dgemv (flip, 1.0, m, v, 0.0, out))\n return out;\n}\n\n/** A convenience function for dot products, which requires less prep and typing than the gsl_cblas_dgexx functions.\n\nIt makes use of the semi-overloading of the \\ref apop_data structure. \\c d1 may be a vector or a matrix, and the same for \\c d2, so this function can do vector dot matrix, matrix dot matrix, and so on. If \\c d1 includes both a vector and a matrix, then later parameters will indicate which to use.\n\n\\param d1 the left part of \\f$ d1 \\cdot d2\\f$\n\\param d2 the right part of \\f$ d1 \\cdot d2\\f$\n\\param form1 't' or 'p': transpose or prime \\c d1->matrix, or, if \\c d1->matrix is \\c NULL, read \\c d1->vector as a row vector.
\n 'n' or 0: use matrix if present; no transpose. (the default)
\n 'v': ignore the matrix and use the vector.\n\n\\param form2 As above, with \\c d2.\n\\return an \\ref apop_data set. If two matrices come in, the vector element is \\c NULL and the \n matrix has the dot product; if either or both are vectors,\n the vector has the output and the matrix is \\c NULL.\n\n\\exception out->error='a' Allocation error.\n\\exception out->error='d' dimension-matching error.\n\\exception out->error='m' GSL math error.\n\\exception NULL If you ask me to take the dot product of NULL, I return NULL.\n\n\\li Some systems auto-transpose non-conforming matrices. You input a \\f$3 \\times 5\\f$ and\na \\f$3 \\times 5\\f$ matrix, and the system assumes that you meant to transpose the second,\nproducing a \\f$(3 \\times 5) \\cdot (5 \\times 3) \\rightarrow (3 \\times 3)\\f$ output. Apophenia\ndoes not do this. First, it's ambiguous whether the output should be \\f$3 \\times 3\\f$\nor \\f$5 \\times 5\\f$. Second, your next run might have three observations, and two \\f$3 \\times 3\\f$ \nmatrices don't require transposition; auto-transposition thus creates situations where\nbugs can pop up on only some iterations of a loop.\n\\li For a vector \\f$\\cdot\\f$ a matrix, the vector is always treated as a row vector,\nmeaning that a \\f$(3\\times 1)\\f$ dot a \\f$(3\\times 4)\\f$ matrix is correct, and produces a\n\\f$(1 \\times 4)\\f$ vector. For a matrix \\f$\\cdot\\f$ a vector, the vector is always treated\nas a column vector. Requests for transposing the vector are ignored in both cases. \n\\li As a corrollary to the above rule, a vector dot a vector always produces a scalar,\n which will be put in the zeroth element of the output vector;\nsee the example. \n\\li If you want to multiply an \\f$N \\times 1\\f$ vector \\f$\\cdot\\f$ a \\f$1 \\times N\\f$\nvector to produce an \\f$N \\times N\\f$ matrix, then use \\ref apop_vector_to_matrix to turn\nyour vectors into matrices; see the example.\n\\li A note for readers of Modeling with Data: the awkward instructions on using\nthis function on p 130 are now obsolete, thanks to the designated initializer syntax\nfor function calls. Notably, in the case where d1 is a vector and d2\na matrix, then apop_dot(d1,d2,'t') won't work, because 't' now refers\nto d1. Instead use apop_dot(d1,d2,.form2='t') or apop_dot(d1,d2,0,\n't')\n\\li This function uses the \\ref designated syntax for inputs.\n\nSample code:\n\\include dot_products.c\n*/\nAPOP_VAR_HEAD apop_data * apop_dot(const apop_data *d1, const apop_data *d2, char form1, char form2){\n const apop_data * apop_varad_var(d1, NULL)\n const apop_data * apop_varad_var(d2, NULL)\n Apop_stopif(!d1, return NULL, 1, \"d1 is NULL; returning NULL\");\n Apop_stopif(!d2, return NULL, 1, \"d2 is NULL; returning NULL\");\n char apop_varad_var(form1, 0)\n char apop_varad_var(form2, 0)\nAPOP_VAR_ENDHEAD\n Set_gsl_handler\n int uselm, userm;\n gsl_matrix *lm = d1->matrix, \n *rm = d2->matrix;\n gsl_vector *lv = d1->vector, \n *rv = d2->vector;\n\n if (d1->matrix && form1 != 'v') uselm = 1;\n else if (d1->vector) uselm = 0;\n else {\n Apop_stopif(form1 == 'v', return NULL, 0,\n \"You asked for a vector from the left data set, but \"\n \"its vector==NULL. Returning NULL.\");\n Apop_stopif(1, return NULL, 0, \"The left data set has neither non-NULL \"\n \"matrix nor vector. Returning NULL.\");\n }\n if (d2->matrix && form2 != 'v') userm = 1;\n else if (d2->vector) userm = 0;\n else {\n Apop_stopif(form2 == 'v', return NULL, 0, \n \"You asked for a vector from the right data set, but \"\n \"its vector==NULL. Returning NULL.\");\n Apop_stopif(1, return NULL, 0, \"The right data set has neither non-NULL \"\n \"matrix nor vector. Returning NULL.\");\n }\n apop_data *out = apop_data_alloc();\n #define Dimcheck(lr, lc, rr, rc) Apop_stopif((lc)!=(rr), out->error='d'; goto done,\\\n 0, \"mismatched dimensions: %zuX%zu dot %zuX%zu. %s\", (lr), (lc), (rr), (rc),\\\n ((lr)==(rr)) ? \" Maybe transpose the first?\" \\\n : ((rc)==(lc)) ? \" Maybe transpose the second?\" : \"\");\n\n CBLAS_TRANSPOSE_t lt, rt;\n lt = (form1 == 'p' || form1 == 't' || form1 == 1) \n ? CblasTrans: CblasNoTrans;\n rt = (form2 == 'p' || form2 == 't' || form2 == 1) \n ? CblasTrans: CblasNoTrans;\n if (uselm && userm){\n Dimcheck((lt== CblasNoTrans) ? lm->size1:lm->size2,\n (lt== CblasNoTrans) ? lm->size2:lm->size1,\n (rt== CblasNoTrans) ? rm->size1:rm->size2,\n (rt== CblasNoTrans) ? rm->size2:rm->size1)\n gsl_matrix *outm = gsl_matrix_calloc((lt== CblasTrans)? lm->size2: lm->size1, \n (rt== CblasTrans)? rm->size1: rm->size2);\n Check_gsl_with_out(gsl_blas_dgemm (lt,rt, 1, lm, rm, 0, outm))\n out->matrix = outm;\n } else if (!uselm && userm){\n Dimcheck((size_t)1, lv->size,\n (rt== CblasNoTrans) ? rm->size1:rm->size2,\n (rt== CblasNoTrans) ? rm->size2:rm->size1)\n //dgemv is always matrix first, then vector, so reverse from vm to mv:\n // if output vector has dimension matrix->size2, send CblasTrans\n // if output vector has dimension matrix->size1, send CblasNoTrans\n out->vector = dot_for_apop_dot(rm, lv\n , (rt == CblasNoTrans) ? CblasTrans : CblasNoTrans);\n Apop_stopif(!out->vector, out->error='m'; goto done, 0, \"GSL-level math error\");\n } else if (uselm && !userm){\n Dimcheck((lt== CblasNoTrans) ? lm->size1:lm->size2,\n (lt== CblasNoTrans) ? lm->size2:lm->size1,\n rv->size , (size_t)1)\n out->vector = dot_for_apop_dot(lm, rv , lt);\n Apop_stopif(!out->vector, out->error='m'; goto done, 0, \"GSL-level math error\");\n } else if (!uselm && !userm){ \n double outd;\n Check_gsl_with_out(gsl_blas_ddot(lv, rv, &outd))\n out->vector = gsl_vector_alloc(1);\n gsl_vector_set(out->vector, 0, outd);\n }\n\n //If using the vector, there's no meaningful name to assign.\n if (d1->names && uselm){\n if (lt == CblasTrans) apop_name_stack(out->names, d1->names, 'r', 'c');\n else apop_name_stack(out->names, d1->names, 'r');\n }\n if (d2->names && userm){\n if (rt == CblasTrans) apop_name_stack(out->names, d2->names, 'c', 'r');\n else apop_name_stack(out->names, d2->names, 'c');\n }\n\ndone:\n Unset_gsl_handler\n return out;\n}\n" }, { "path": "apop_linear_constraint.m4.c", "content": "/** \\file apop_linear_constraint.c \n \\c apop_linear_constraint finds a point that meets a set of linear constraints. This takes a lot of machinery, so it gets its own file.\n\nCopyright (c) 2007, 2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. \n*/\n#include \"apop_internal.h\"\n\nstatic double magnitude(gsl_vector *v){\n double out;\n gsl_blas_ddot(v, v, &out);\n return out;\n}\n\nstatic void find_nearest_point(gsl_vector *V, double k, gsl_vector *B, gsl_vector *out){\n /* Find X such that BX =K and there is an S such that X + SB=V. */\n double S=0; //S = (BV-K)/B'B.\n gsl_blas_ddot(B, V, &S);\n S -= k;\nassert(!gsl_isnan(S));\n S /= magnitude(B);\nassert(!gsl_isnan(S));\n gsl_vector_memcpy(out, B); //X = -SB +V\n gsl_vector_scale(out, -S);\n gsl_vector_add(out, V);\nassert(!gsl_isnan(gsl_vector_get(out,0)));\n}\n\nstatic int binds(gsl_vector const *v, double k, gsl_vector const *b, double margin){\n double d;\n gsl_blas_ddot(v, b, &d);\n return d < k + margin;\n}\n\nstatic double trig_bit(gsl_vector *dimv, gsl_vector *otherv, double off_by){\n double theta, costheta, dot, out;\n gsl_blas_ddot(dimv, otherv, &dot);\n costheta = dot/(magnitude(dimv)*magnitude(otherv));\n theta = acos(costheta);\n out = off_by/gsl_pow_2(sin(theta)); \n return out;\n}\n\n/* The hard part is when your candidate point does not satisfy other\n constraints, so you need to translate the point until it meets the new hypersurface.\n How far is that? Project beta onto the new surface, and find the\n distance between that projection and the original surface. Then\n translate beta toward the original surface by that amount. The\n projection of the translated beta onto the new surface now also touches the old\n surface.\n */\nstatic void get_candiate(gsl_vector *beta, apop_data *constraint, int current, gsl_vector *candidate, double margin){\n double k, ck, off_by, s;\n gsl_vector *pseudobeta = NULL;\n gsl_vector *pseudocandidate = NULL;\n gsl_vector *pseudocandidate2 = NULL;\n gsl_vector *fix = NULL;\n gsl_vector *cc = Apop_rv(constraint, current);\n ck = gsl_vector_get(constraint->vector, current);\n find_nearest_point(beta, ck, cc, candidate);\n for (size_t i=0; i< constraint->vector->size; i++){\n if (i!=current){\n gsl_vector *other = Apop_rv(constraint, i);\n k =apop_data_get(constraint, i, -1);\n if (binds(candidate, k, other, margin)){\n if (!pseudobeta){\n pseudobeta = gsl_vector_alloc(beta->size);\n gsl_vector_memcpy(pseudobeta, beta);\n pseudocandidate = gsl_vector_alloc(beta->size);\n pseudocandidate2 = gsl_vector_alloc(beta->size);\n fix = gsl_vector_alloc(beta->size);\n }\n find_nearest_point(pseudobeta, k, other, pseudocandidate);\n find_nearest_point(pseudocandidate, ck, cc, pseudocandidate2);\n off_by = apop_vector_distance(pseudocandidate, pseudocandidate2);\n s = trig_bit(cc, other, off_by);\n gsl_vector_memcpy(fix, cc);\n gsl_vector_scale(fix, magnitude(cc));\n gsl_vector_scale(fix, s);\n gsl_vector_add(pseudobeta, fix);\n find_nearest_point(pseudobeta, k, other, candidate);\n gsl_vector_memcpy(pseudobeta, candidate);\n } \n }\n }\n if (fix){ \n gsl_vector_free(fix); gsl_vector_free(pseudobeta);\n gsl_vector_free(pseudocandidate); gsl_vector_free(pseudocandidate2);\n }\n}\n\n/** This is designed to be called from within the constraint method of your \\ref\napop_model. Just write the constraint vector+matrix and this will do the rest.\nSee \\ref constr for detailed discussion. \n \n\\param beta The proposed vector about to be tested. No default, must not be \\c NULL.\n\n\\param constraint \nA vector/matrix pair [v | m1 m2 ... mn] where each row is interpreted as a less-than inequality:\n\\f$v < m1x1+ m2x2 + ... + mnxn\\f$. For example, say your constraints are \n\\f$3 < 2x + 4y - 7z\\f$ and \\f$y\\f$ is positive, i.e. \\f$0 < y\\f$.\nAllocate and fill the matrix representing these two constraints via:\n\\code\napop_data *constr = apop_data_falloc((2,2,3), 3, 2, 4, 7,\n 0, 0, 1, 0);\n\\endcode\n. Default: each elements is greater than zero. For three parameters this would be equivalent to setting\n\\code\napop_data *constr = apop_data_falloc((3,3,3), 0, 1, 0, 0,\n 0, 0, 1, 0,\n 0, 0, 0, 1);\n\\endcode\n\n\\param margin If zero, then this is a >= constraint, otherwise I will return a point this amount within the borders. You could try \\c GSL_DBL_EPSILON, which is the smallest value a \\c double can hold, or something like 1e-3. Default = 0.\n\n\\return The penalty: the distance between beta and the closest point that meets the constraints.\nIf the constraint is met, the penalty is zero.\nIf the constraint is not met, this \\c beta is shifted by \\c margin (Euclidean distance) to meet the constraints. \n\n \\li If your \\ref apop_data has more structure than a vector, try \\ref apop_data_pack to pack it\ninto a vector. This is what \\ref apop_maximum_likelihood does.\n \\li The function doesn't check for odd cases like coplanar constraints.\n \\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD long double apop_linear_constraint(gsl_vector *beta, apop_data * constraint, double margin){\n static threadlocal apop_data *default_constraint;\n gsl_vector * apop_varad_var(beta, NULL);\n double apop_varad_var(margin, 0);\n apop_data * apop_varad_var(constraint, NULL);\n Apop_assert(beta, \"The vector to be checked is NULL.\");\n if (!constraint){\n if (default_constraint && beta->size != default_constraint->vector->size){\n apop_data_free(default_constraint);\n default_constraint = NULL;\n }\n if (!default_constraint){\n default_constraint = apop_data_alloc(0,beta->size, beta->size);\n default_constraint->vector = gsl_vector_calloc(beta->size);\n gsl_matrix_set_identity(default_constraint->matrix);\n }\n constraint = default_constraint;\n }\nAPOP_VAR_ENDHEAD\n static threadlocal gsl_vector *closest_pt = NULL;\n static threadlocal gsl_vector *candidate = NULL;\n static threadlocal gsl_vector *fix = NULL;\n int constraint_ct = constraint->matrix->size1;\n int bindlist[constraint_ct];\n int i, bound = 0;\n /* For added efficiency, keep a scratch vector or two on hand. */\n if (closest_pt==NULL || closest_pt->size != constraint->matrix->size2){\n closest_pt = gsl_vector_calloc(beta->size);\n candidate = gsl_vector_alloc(beta->size);\n fix = gsl_vector_alloc(beta->size);\n closest_pt->data[0] = GSL_NEGINF;\n }\n /* Do any constraints bind?*/\n memset(bindlist, 0, sizeof(int)*constraint_ct);\n for (i=0; i< constraint_ct; i++){\n gsl_vector *c = Apop_rv(constraint, i);\n bound +=\n bindlist[i] = binds(beta, apop_data_get(constraint, i, -1), c, margin);\n }\n if (!bound) return 0; //All constraints met.\n gsl_vector *base_beta = apop_vector_copy(beta);\n /* With only one constraint, it's easy. */\n if (constraint->vector->size==1){\n gsl_vector *c = Apop_rv(constraint, 0);\n find_nearest_point(base_beta, constraint->vector->data[0], c, beta);\n goto add_margin;\n }\n /* Finally, multiple constraints, at least one binding.\n For each surface, pick a candidate point.\n Check whether the point meets the other constraints. \n if not, translate to a new point that works.\n [Do this by maintaining a pseudopoint that translates by the\n necessary amount.]\n Once you have a candidate point, compare its distance to the\n current favorite; keep the best.\n */\n for (i=0; i< constraint_ct; i++)\n if (bindlist[i]){\n get_candiate(base_beta, constraint, i, candidate, margin);\n if(apop_vector_distance(base_beta, candidate) < apop_vector_distance(base_beta, closest_pt))\n gsl_vector_memcpy(closest_pt, candidate);\n }\n gsl_vector_memcpy(beta, closest_pt);\nadd_margin:\n for (i=0; i< constraint_ct; i++){\n if(bindlist[i]){\n gsl_vector_memcpy(fix, Apop_rv(constraint, i));\n gsl_vector_scale(fix, magnitude(fix));\n gsl_vector_scale(fix, margin);\n gsl_vector_add(beta, fix);\n }\n }\n long double out = apop_vector_distance(base_beta, beta);\n gsl_vector_free(base_beta);\n return out;\n}\n" }, { "path": "apop_mapply.m4.c", "content": "/** \\file apop_mapply.c vector/matrix map/apply. */\n/* Copyright (c) 2007, 2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. \n \n This file is a tour de force of the if statement. There are several possibilities:\n\n --user wants the vector, matrix, rows, columns, all items in the data set\n --user wants output in a new location, or written to the old.\n --user has extra parameters\n --user needs to know the index of the function\n --user wants the sum of the result (e.g., to find how many elements are NAN, or a sum of log-likelihoods).\n\n Further, Apophenia v0.22 introduced variadic, optional arguments, so\n we have a somewhat more robust syntax post-22, and also the prior syntax,\n which some may find useful.\n\n We thus have a lot of functions that all feed in to mapply_core, which, after several if statements,\n then dispatches segments do different threads, and either the vectorloop or forloop that does all the\n actual math.\n\n */\n#include \"apop_internal.h\"\n#include \nstatic gsl_vector*mapply_core(apop_data *d, gsl_matrix *m, gsl_vector *vin, void *fn, gsl_vector *vout, bool use_index, bool use_param,void *param, char post_22, bool by_apop_rows);\n\ntypedef double apop_fn_v(gsl_vector*);\ntypedef void apop_fn_vtov(gsl_vector*);\ntypedef double apop_fn_d(double);\ntypedef void apop_fn_dtov(double*);\ntypedef double apop_fn_r(apop_data*);\ntypedef double apop_fn_vp(gsl_vector*, void *);\ntypedef double apop_fn_dp(double, void *);\ntypedef double apop_fn_rp(apop_data*, void *);\ntypedef double apop_fn_vpi(gsl_vector*, void *, int);\ntypedef double apop_fn_dpi(double, void *, int);\ntypedef double apop_fn_rpi(apop_data*, void *, int);\ntypedef double apop_fn_vi(gsl_vector*, int);\ntypedef double apop_fn_di(double, int);\ntypedef double apop_fn_ri(apop_data*, int);\n\n\n/** Apply a function to every element of a data set, matrix or vector; or, apply a\nvector-taking function to every row or column of a matrix.\n\nYour function could take any combination of a \\c gsl_vector, a \\c double, an \\ref apop_data, a parameter set, and the position of the element in the vector or matrix. As such, the function takes twelve function inputs, one for each combination of vector/matrix, params/no params, index/no index. Fortunately, because \nthis function uses the \\ref designated syntax for inputs, you will specify only one.\n\nFor example, here is a function that will cut off each element of the input data to\nbetween \\f$(-1, +1)\\f$. It takes in a lone \\c double and a parameter in a \\c void*,\nso it gets sent to \\ref apop_map via .fn_dp=cutoff.\n\\code\ndouble cutoff(double in, void *limit_in){ \n double *limit = limit_in;\n return GSL_MAX(-*limit, GSL_MIN(*limit, in)); \n}\n\ndouble param = 1;\napop_map(your_data, .fn_dp=cutoff, .param=¶m, .inplace='y');\n\\endcode\n\n\\param fn_v A function of the form double your_fn(gsl_vector *in)\n\\param fn_d A function of the form double your_fn(double in)\n\\param fn_r A function of the form double your_fn(apop_data *in)\n\\param fn_vp A function of the form double your_fn(gsl_vector *in, void *param)\n\\param fn_dp A function of the form double your_fn(double in, void *param)\n\\param fn_rp A function of the form double your_fn(apop_data *in, void *param)\n\\param fn_vpi A function of the form double your_fn(gsl_vector *in, void *param, int index)\n\\param fn_dpi A function of the form double your_fn(double in, void *param, int index)\n\\param fn_rpi A function of the form double your_fn(apop_data *in, void *param, int index)\n\\param fn_vi A function of the form double your_fn(gsl_vector *in, int index)\n\\param fn_di A function of the form double your_fn(double in, int index)\n\\param fn_ri A function of the form double your_fn(apop_data *in, int index)\n\n\\param in The input data set. If \\c NULL, I'll return \\c NULL immediately.\n\\param param A pointer to the parameters to be passed to those function forms taking a \\c *param.\n\n\\param part Which part of the \\c apop_data struct should I use?
\n'v'==Just the vector
\n'm'==Every element of the matrix, in turn
\n'a'==Both 'v' and 'm'
\n'r'==Apply a function \\c gsl_vector \\f$\\to\\f$ \\c double to each row of the matrix
\n'c'==Apply a function \\c gsl_vector \\f$\\to\\f$ \\c double to each column of the matrix
\nDefault is 'a', but notice that I'll ignore a \\c NULL vector or matrix, so if your data set has only a vector or only a matrix, that's what I'll use.\n\n\\param all_pages If \\c 'y', then follow the \\c more pointer to subsequent pages. If \\c 'n',\n handle only the first page of data. Default: \\c 'n'. \n\n\\param inplace If 'n' (the default), generate a new \\ref apop_data set for output,\nwhich will contain the mapped values (and the names from the original set).
\nIf 'y',\nmodify in place. The \\c double \\f$\\to\\f$ \\c double versions, \\c 'v', \\c 'm', and \\c\n'a', write to exactly the same location as before. The \\c gsl_vector \\f$\\to\\f$ \\c\ndouble versions, \\c 'r', and \\c 'c', will write to the vector. Be careful: if you\nare writing in place and there is already a vector there, then the original vector is\nlost.
\nIf 'v' (as in void), return \\c NULL. (Default = 'n')\n\n\\exception out->error='p' missing or mismatched parts error, such as \\c NULL matrix when you sent a function acting on the matrix element.\n\n\\li The function forms with r in them, like \\c fn_ri, are row-by-row. I'll use\n\\ref Apop_r to get each row in turn, and send it to the function. The first\nimplication is that your function should be expecting a \\ref apop_data set with\nexactly one row in it. The second is that \\c part is ignored: it only makes sense to go\nrow-by-row. \n \\li For these \\c r functions, if you set \\c inplace='y', then you will be modifying\nyour input data set, row by row; if you set \\c inplace='n', then I will return an \\ref\napop_data set whose \\c vector element is as long as your data set (i.e., as long as\nthe longest of your text, vector, or matrix parts).\n \\li If you set omp_set_num_threads(n) using \\f$n>1\\f$,\nsplit the data set into as many chunks as you specify and process them\nsimultaneously. You need to watch out for the usual hang-ups about multithreaded\nprogramming, but if your data is iid, and each row's processing is independent of the\nothers, you should have no problems. Bear in mind that generating threads takes some\nsmall overhead, so simple cases like adding a few hundred numbers will actually be\nslower when threading.\n \\li See \\ref mapply for many more examples and notes.\n\\see apop_map_sum\n\\ingroup all_public\n*/\nAPOP_VAR_HEAD apop_data* apop_map(apop_data *in, apop_fn_d *fn_d, apop_fn_v *fn_v, apop_fn_r *fn_r, apop_fn_dp *fn_dp, apop_fn_vp *fn_vp, apop_fn_rp *fn_rp, apop_fn_dpi *fn_dpi, apop_fn_vpi *fn_vpi, apop_fn_rpi *fn_rpi, apop_fn_di *fn_di, apop_fn_vi *fn_vi, apop_fn_ri *fn_ri, void *param, int inplace, char part, int all_pages){ \n apop_data * apop_varad_var(in, NULL)\n if (!in) return NULL;\n apop_fn_v * apop_varad_var(fn_v, NULL)\n apop_fn_d * apop_varad_var(fn_d, NULL)\n apop_fn_r * apop_varad_var(fn_r, NULL)\n apop_fn_vp * apop_varad_var(fn_vp, NULL)\n apop_fn_dp * apop_varad_var(fn_dp, NULL)\n apop_fn_rp * apop_varad_var(fn_rp, NULL)\n apop_fn_vpi * apop_varad_var(fn_vpi, NULL)\n apop_fn_dpi * apop_varad_var(fn_dpi, NULL)\n apop_fn_rpi * apop_varad_var(fn_rpi, NULL)\n apop_fn_vi * apop_varad_var(fn_vi, NULL)\n apop_fn_di * apop_varad_var(fn_di, NULL)\n apop_fn_ri * apop_varad_var(fn_ri, NULL)\n int apop_varad_var(inplace, 'n')\n void * apop_varad_var(param, NULL)\n int by_vectors = fn_v || fn_vp || fn_vpi || fn_vi;\n char apop_varad_var(part, by_vectors ? 'r' : 'a')\n int apop_varad_var(all_pages, 'n')\nAPOP_VAR_ENDHEAD\n int use_param = (fn_vp || fn_dp || fn_rp || fn_vpi || fn_rpi || fn_dpi);\n int use_index = (fn_vi || fn_di || fn_ri || fn_vpi || fn_rpi|| fn_dpi);\n //Give me the first non-null input function.\n void *fn = fn_v ? (void *)fn_v : fn_d ? (void *)fn_d : fn_r ? (void *)fn_r : fn_vp ? (void *)fn_vp : fn_dp ? (void *)fn_dp :fn_rp ? (void *)fn_rp : fn_vpi ? (void *)fn_vpi : fn_rpi ? (void *)fn_rpi: fn_dpi ? (void *)fn_dpi : fn_vi ? (void *)fn_vi : fn_di ? (void *)fn_di : fn_ri ? (void *)fn_ri : NULL;\n\n int by_apop_rows = fn_r || fn_rp || fn_rpi || fn_ri;\n\n Apop_stopif((part=='c' || part=='r') && (fn_d || fn_dp || fn_dpi || fn_di), \n apop_return_data_error(p),\n 0, \"You asked for a vector-oriented operation (.part='r' or .part='c'), but \"\n \"gave me a scalar-oriented function. Did you mean part=='a'?\");\n\n //Allocate output\n Get_vmsizes(in); //vsize, msize1, msize2, maxsize\n apop_data *out = (inplace=='y') ? in\n : (inplace=='v') ? NULL\n : by_apop_rows ? apop_data_alloc(GSL_MAX(in->textsize[0], maxsize))\n : part == 'v' || (in->vector && ! in->matrix) ? apop_data_alloc(vsize)\n : part == 'm' ? apop_data_alloc(msize1, msize2)\n : part == 'a' ? apop_data_alloc(vsize, msize1, msize2)\n : part == 'r' ? apop_data_alloc(maxsize)\n : part == 'c' ? apop_data_alloc(msize2) : NULL;\n Apop_stopif(inplace=='y' && (part=='r'||part=='c') && !in->vector, in->vector=gsl_vector_alloc(maxsize), 2, \n \"No vector in your input data set for me to write outputs to; \"\n \"allocating one for you of size %i\", maxsize);\n if (in->names && out && !(inplace=='y')){\n if (part == 'v' || (in->vector && ! in->matrix)) {\n apop_name_stack(out->names, in->names, 'v');\n apop_name_stack(out->names, in->names, 'r');\n }\n else if (part == 'm'){\n apop_name_stack(out->names, in->names, 'r');\n apop_name_stack(out->names, in->names, 'c');\n }\n else if (!by_apop_rows && part == 'a'){\n apop_name_free(out->names);\n out->names = apop_name_copy(in->names);\n } else if (by_apop_rows || part == 'r')\n apop_name_stack(out->names, in->names, 'r');\n else if (part == 'c')\n apop_name_stack(in->names, out->names, 'r', 'c');\n }\n\n if (by_apop_rows) mapply_core(in, NULL, NULL, fn, out ? out->vector : NULL, use_index, use_param, param, 'r', by_apop_rows);\n else {\n if (in->vector && (part == 'v' || part=='a'))\n mapply_core(NULL, NULL, in->vector, fn, out ? out->vector : NULL, use_index, use_param, param, 'r', by_apop_rows);\n if (in->matrix && (part == 'm' || part=='a')){\n int smaller_dim = GSL_MIN(in->matrix->size1, in->matrix->size2);\n for (int i=0; i< smaller_dim; i++){\n if (smaller_dim == in->matrix->size1){\n gsl_vector *onevector = Apop_rv(in, i);\n if (inplace=='v')\n mapply_core(NULL, NULL, onevector, fn, NULL, use_index, use_param, param, 'r', by_apop_rows);\n else mapply_core(NULL, NULL, onevector, fn, Apop_rv(out, i), use_index, use_param, param, 'r', by_apop_rows);\n } else {\n gsl_vector *onevector = Apop_cv(in, i);\n if (inplace=='v')\n mapply_core(NULL, NULL, onevector, fn, NULL, use_index, use_param, param, 'c', by_apop_rows);\n else {\n gsl_vector *twovector = Apop_cv(out, i);\n mapply_core(NULL, NULL, onevector, fn, twovector, use_index, use_param, param, 'c', by_apop_rows);\n }\n }\n }\n }\n if (part == 'r' || part == 'c'){\n Apop_stopif(!in->matrix, if (!out) out=apop_data_alloc(); out->error='p'; return out,\n 0, \"You asked for me to operate on the %cs of the matrix, but the matrix is NULL.\", part);\n mapply_core(NULL, in->matrix, NULL, fn, out ? out->vector : NULL, use_index, use_param, param, part, by_apop_rows);\n }\n }\n if ((all_pages=='y' || all_pages=='Y') && in->more){\n out->more = apop_map_base(in->more, fn_d, fn_v, fn_r, fn_dp, fn_vp, fn_rp, fn_dpi, fn_vpi, fn_rpi, fn_di, fn_vi, fn_ri, param, inplace, part, all_pages);\n Apop_stopif(out->more->error, out->error=out->more->error, 1, \"Error in subpage; marked parent page with same error code.\");\n }\n return out;\n}\n\n/** \\cond doxy_ignore */\ntypedef struct {\n void *fn;\n gsl_matrix *m;\n gsl_vector *v, *vin;\n apop_data *d;\n bool use_index, use_param;\n char rc;\n void *param;\n} threadpass;\n/** \\endcond */\n\n/* Mapply_core splits the database into an array of threadpass structs, then one of the following\n ...loop functions gets called, which does the actual for loop to step through the rows/columns/elements. */\n\nstatic void rowloop(threadpass *tc){\n apop_fn_r *rtod=tc->fn;\n apop_fn_rp *fn_rp=tc->fn;\n apop_fn_rpi *fn_rpi=tc->fn;\n apop_fn_ri *fn_ri=tc->fn;\n Get_vmsizes(tc->d); //maxsize\n OMP_for (int i=0; i< maxsize; i++){\n apop_data *onerow = Apop_r(tc->d, i);\n double val = \n tc->use_param ? (tc->use_index ? fn_rpi(onerow, tc->param, i) : fn_rp(onerow, tc->param) )\n : (tc->use_index ? fn_ri(onerow, i) : rtod(onerow) );\n if (tc->v) gsl_vector_set(tc->v, i, val);\n }\n}\n\nstatic void forloop(threadpass *tc){\n apop_fn_v *vtod=tc->fn;\n apop_fn_vp *fn_vp=tc->fn;\n apop_fn_vpi *fn_vpi=tc->fn;\n apop_fn_vi *fn_vi=tc->fn;\n int max = tc->rc == 'r' ? tc->m->size1 : tc->m->size2;\n OMP_for (int i= 0; i< max; i++){\n gsl_vector view = tc->rc == 'r' ? gsl_matrix_row(tc->m, i).vector : gsl_matrix_column(tc->m, i).vector;\n double val = \n tc->use_param ? (tc->use_index ? fn_vpi(&view, tc->param, i) : fn_vp(&view, tc->param) )\n : (tc->use_index ? fn_vi(&view, i) : vtod(&view) );\n if (tc->v) gsl_vector_set(tc->v, i, val);\n }\n}\n\nstatic void oldforloop(threadpass *tc){\n apop_fn_vtov *vtov=tc->fn;\n if (tc->v){\n tc->rc = 'r';\n return forloop(tc);\n }\n OMP_for (int i=0; i< tc->m->size1; i++)\n vtov(Apop_mrv(tc->m, i));\n}\n\n//if mapping to self, then set tc.v = in_v\nstatic void vectorloop(threadpass *tc){\n apop_fn_d *dtod=tc->fn;\n apop_fn_dp *fn_dp=tc->fn;\n apop_fn_dpi *fn_dpi=tc->fn;\n apop_fn_di *fn_di=tc->fn;\n OMP_for (int i= 0; i< tc->vin->size; i++){\n double inval = gsl_vector_get(tc->vin, i);\n double outval =\n tc->use_param ? (tc->use_index ? fn_dpi(inval, tc->param, i) : \n fn_dp(inval, tc->param))\n : (tc->use_index ? fn_di(inval, i) : \n dtod(inval));\n if (tc->v) gsl_vector_set(tc->v, i, outval);\n }\n}\n\nstatic void oldvectorloop(threadpass *tc){\n apop_fn_dtov *dtov=tc->fn;\n if (tc->v) return vectorloop(tc);\n OMP_for (int i= 0; i< tc->vin->size; i++){\n double *inval = gsl_vector_ptr(tc->vin, i);\n dtov(inval);\n }\n}\n\nstatic gsl_vector*mapply_core(apop_data *d, gsl_matrix *m, gsl_vector *vin, void *fn, gsl_vector *vout, bool use_index, bool use_param, void *param, char post_22, bool by_apop_rows){\n Get_vmsizes(d); //maxsize\n threadpass tp =\n (threadpass) {\n .fn = fn, .m = m, .d = d,\n .vin = vin, .v = vout,\n .use_index = use_index, .use_param= use_param,\n .param = param, .rc = post_22\n };\n if (by_apop_rows) rowloop(&tp);\n else if (m) post_22 ? forloop(&tp) : oldforloop(&tp);\n else post_22 ? vectorloop(&tp) : oldvectorloop(&tp);\n return vout;\n}\n\n/** Map a function onto every row of a matrix. The function that you input takes in a\n\\c gsl_vector and returns a \\c double. This function will produce a sequence of vector\nviews of each row of the input matrix, and send each to your function. It will output\na \\c gsl_vector holding your function's output for each row.\n\n \\param m The matrix\n \\param fn A function of the form double fn(gsl_vector* in)\n\n \\return A \\c gsl_vector with the corresponding value for each row.\n\n \\li If you input a \\c NULL matrix, I return \\c NULL.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\ngsl_vector *apop_matrix_map(const gsl_matrix *m, double (*fn)(gsl_vector*)){\n if (!m) return NULL;\n gsl_vector *out = gsl_vector_alloc(m->size1);\n return mapply_core(NULL, (gsl_matrix*) m, NULL, fn, out, 0, 0, NULL, 0, false);\n}\n\n/** Apply a function to every row of a matrix. The function that you input takes in\na \\c gsl_vector and returns nothing. \\c apop_matrix_apply will produce a vector view of\neach row, and send each row to your function.\n\n \\param m The matrix\n \\param fn A function of the form void fn(gsl_vector* in) which may modify\n the data at the \\c in pointer in place.\n\n \\li If the matrix is \\c NULL, this is a no-op and returns immediately.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\nvoid apop_matrix_apply(gsl_matrix *m, void (*fn)(gsl_vector*)){\n if (!m) return;\n mapply_core(NULL, m, NULL, fn, NULL, 0, 0, NULL, 0, false);\n}\n\n/** Map a function onto every element of a vector. Thus function will send each\nelement to the function you provide, and will output a \\c gsl_vector holding your\nfunction's output for each row.\n\n \\param v The input vector\n \\param fn A function of the form double fn(double in)\n\n \\return A \\c gsl_vector (allocated by this function) with the corresponding value for each row.\n\n \\li If you input a \\c NULL vector, I return \\c NULL.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\ngsl_vector *apop_vector_map(const gsl_vector *v, double (*fn)(double)){\n if (!v) return NULL;\n gsl_vector *out = gsl_vector_alloc(v->size);\n return mapply_core(NULL, NULL, (gsl_vector*) v, fn, out, 0, 0, NULL, 0, false);\n}\n\n/** Apply a function to every row of a matrix. The function that you input takes in\na \\c double* and may modify the input value in place. This function will send a pointer\nto each element of your vector to your function.\n\n \\param v The input vector\n \\param fn A function of the form void fn(double in)\n\n \\li If the vector is \\c NULL, this is a no-op.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map\n*/\nvoid apop_vector_apply(gsl_vector *v, void (*fn)(double*)){\n if (!v) return;\n mapply_core(NULL, NULL, v, fn, NULL, 0, 0, NULL, 0, false); }\n\nstatic void apop_matrix_map_all_vector_subfn(const gsl_vector *in, gsl_vector *outv, double (*fn)(double)){\n mapply_core(NULL, NULL, (gsl_vector *) in, fn, outv, 0, 0, NULL, 0, false); }\n\n/** Maps a function to every element in a matrix (as opposed to every row).\n\n \\param in The matrix whose elements will be inputs to the function\n \\param fn A function with a form like double f(double in).\n \\return a matrix of the same size as the original, with the function applied.\n\n \\li If you input a \\c NULL matrix, I return \\c NULL.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\n\ngsl_matrix * apop_matrix_map_all(const gsl_matrix *in, double (*fn)(double)){\n if (!in) return NULL;\n gsl_matrix *out = gsl_matrix_alloc(in->size1, in->size2);\n OMP_for (size_t i=0; i< in->size1; i++){\n gsl_vector_const_view inv = gsl_matrix_const_row(in, i);\n apop_matrix_map_all_vector_subfn(&inv.vector, Apop_mrv(out, i), fn);\n }\n return out;\n}\n\n/** Applies a function to every element in a matrix (as opposed to every row)\n\n \\param in The matrix whose elements will be inputs to the function\n \\param fn A function with a form like void f(double *in) which may modify\n the data at the \\c in pointer in place.\n\n \\li If the matrix is \\c NULL, this is a no-op and returns immediately.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\nvoid apop_matrix_apply_all(gsl_matrix *in, void (*fn)(double *)){\n if (!in) return;\n OMP_for (size_t i=0; i< in->size1; i++){\n apop_vector_apply(Apop_mrv(in, i), fn);\n }\n}\n\n/** Returns the sum of the output of \\c apop_vector_map. For example,\napop_vector_map_sum(v, isnan) returns the count of elements of v\nthat are \\c NaN.\n\n \\li If you input a \\c NULL vector, I return the sum of zero items: zero.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\ndouble apop_vector_map_sum(const gsl_vector *in, double(*fn)(double)){\n if (!in) return 0;\n gsl_vector *m = apop_vector_map (in, fn);\n double out = apop_vector_sum(m);\n gsl_vector_free(m);\n return out;\n}\n\n/** Like \\c apop_matrix_map_all, but returns the sum of the resulting mapped function. For example, apop_matrix_map_all_sum(v, isnan) returns the number of elements of m that are \\c NaN.\n\n \\li If you input a \\c NULL matrix, I return the sum of zero items: zero.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\ndouble apop_matrix_map_all_sum(const gsl_matrix *in, double (*fn)(double)){\n if (!in) return 0;\n gsl_matrix *m = apop_matrix_map_all (in, fn);\n double out = apop_matrix_sum(m);\n gsl_matrix_free(m);\n return out;\n}\n\n/** Like \\c apop_matrix_map, but returns the sum of the resulting mapped vector. For example, let \\c log_like be a function that returns the log likelihood of an input vector; then apop_matrix_map_sum(m, log_like) returns the total log likelihood of the rows of \\c m.\n\n \\li If you input a \\c NULL matrix, I return the sum of zero items: zero.\n \\li See \\ref mapply \"the map/apply page\" for details.\n\\see \\ref apop_map, \\ref apop_map_sum\n*/\ndouble apop_matrix_map_sum(const gsl_matrix *in, double (*fn)(gsl_vector*)){\n if (!in) return 0;\n gsl_vector *v = apop_matrix_map (in, fn);\n double out = apop_vector_sum(v);\n gsl_vector_free(v);\n return out;\n}\n\n/** A function that effectively calls \\ref apop_map and returns the sum of the resulting\nelements. Thus, this function returns a \\c double. See the \\ref apop_map page for\ndetails of the inputs, which are the same here, except that \\c inplace doesn't make\nsense---this function will always just add up the input function outputs.\n\n\\li I don't copy the input data to send to your input function. Therefore, if your\nfunction modifies its inputs as a side-effect, your data set will be modified as this\nfunction runs.\n \\li The sum of zero elements is zero, so that is what is returned if the input \\ref\napop_data set is \\c NULL. If apop_opts.verbose >= 2 print a warning.\n \\li See \\ref mapply for many more examples and notes.\n \\li This function uses the \\ref designated syntax for inputs.\n\\ingroup all_public\n*/\nAPOP_VAR_HEAD double apop_map_sum(apop_data *in, apop_fn_d *fn_d, apop_fn_v *fn_v, apop_fn_r *fn_r, apop_fn_dp *fn_dp, apop_fn_vp *fn_vp, apop_fn_rp *fn_rp, apop_fn_dpi *fn_dpi, apop_fn_vpi *fn_vpi, apop_fn_rpi *fn_rpi, apop_fn_di *fn_di, apop_fn_vi *fn_vi, apop_fn_ri *fn_ri, void *param, char part, int all_pages){ \n apop_data * apop_varad_var(in, NULL)\n Apop_stopif(!in, return 0, 2, \"NULL input. Returning zero.\");\n apop_fn_v * apop_varad_var(fn_v, NULL)\n apop_fn_d * apop_varad_var(fn_d, NULL)\n apop_fn_r * apop_varad_var(fn_r, NULL)\n apop_fn_vp * apop_varad_var(fn_vp, NULL)\n apop_fn_dp * apop_varad_var(fn_dp, NULL)\n apop_fn_rp * apop_varad_var(fn_rp, NULL)\n apop_fn_vpi * apop_varad_var(fn_vpi, NULL)\n apop_fn_dpi * apop_varad_var(fn_dpi, NULL)\n apop_fn_rpi * apop_varad_var(fn_rpi, NULL)\n apop_fn_vi * apop_varad_var(fn_vi, NULL)\n apop_fn_di * apop_varad_var(fn_di, NULL)\n apop_fn_ri * apop_varad_var(fn_ri, NULL)\n void * apop_varad_var(param, NULL)\n char apop_varad_var(part, ((fn_v||fn_vp||fn_vpi||fn_vi) ? 'r' : 'a'));\n int apop_varad_var(all_pages, 'n')\nAPOP_VAR_ENDHEAD \n apop_data *mapped = apop_map(in, .fn_d=fn_d, .fn_v=fn_v, .fn_r=fn_r, \n .fn_dp=fn_dp, .fn_vp=fn_vp, .fn_rp=fn_rp, \n .fn_dpi=fn_dpi, .fn_vpi=fn_vpi, .fn_rpi=fn_rpi, \n .fn_di=fn_di, .fn_vi=fn_vi, .fn_ri=fn_ri, \n .param=param, .part=part, .inplace='n', .all_pages='n');\n double outsum = (mapped->vector ? apop_sum(mapped->vector) : 0)\n + (mapped->matrix ? apop_matrix_sum(mapped->matrix) : 0);\n apop_data_free(mapped);\n return outsum + \n (((all_pages=='y' || all_pages=='Y') && in->more) ? \n apop_map_sum_base(in->more, fn_d, fn_v, fn_r, fn_dp, \n fn_vp, fn_rp, fn_dpi, fn_vpi, fn_rpi, fn_di, fn_vi, \n fn_ri, param, part, all_pages) : 0);\n}\n/** \\} */\n" }, { "path": "apop_mcmc.m4.c", "content": "/** \\file \n Markov Chain Monte Carlo. */ \n/* Copyright (c) 2014 by Ben Klemens. Licensed under the GNU GPL v2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\n\n///default step and adapt fns.\n\nstatic void step_to_vector(double const *d, apop_mcmc_proposal_s *ps, apop_mcmc_settings *ms){\n apop_model *m = ps->proposal;\n memcpy(m->parameters->vector->data, d, sizeof(double)*m->parameters->vector->size);\n\n ps->adapt_fn(ps, ms);\n}\n\nint sigma_adapt(apop_mcmc_proposal_s *ps, apop_mcmc_settings *ms){\n apop_model *m = ps->proposal;\n //accept rate. Add 1% * target to numerator; 1% to denominator, to slow early jumps\n double ar = (ps->accept_count + .01*ms->periods *ms->target_accept_rate)\n /(ps->accept_count + ps->reject_count + .01*ms->periods);\n/* double std_dev_scale= (ar > ms->target_accept_rate) \n ? (2 - (1.-ar)/(1.-ms->target_accept_rate))\n : (1/(2-((ar+0.0)/ms->target_accept_rate)));\n */\n double scale = ar/ms->target_accept_rate;\n scale = 1+ (scale-1)/100.;\n //gsl_matrix_scale(m->parameters->matrix, scale > .1? ( scale < 10 ? scale : 10) : .1);\n gsl_matrix_scale(m->parameters->matrix, scale);\n return 0;\n}\n\n/////// apop_mcmc_settings\n\nApop_settings_init(apop_mcmc,\n Apop_varad_set(periods, 6e3);\n Apop_varad_set(burnin, 0.05);\n Apop_varad_set(target_accept_rate, 0.35);\n Apop_varad_set(gibbs_chunks, 'b');\n Apop_varad_set(start_at, '1');\n Apop_varad_set(base_step_fn, step_to_vector);\n Apop_varad_set(base_adapt_fn, sigma_adapt);\n //all else defaults to zero/NULL\n)\n\nApop_settings_copy(apop_mcmc, \n if (in->block_count){\n out->proposals = calloc(in->block_count, sizeof(apop_mcmc_proposal_s));\n for (int i=0; i< in->block_count; i++){\n out->proposals[i] = in->proposals[i];\n out->proposals[i].proposal = apop_model_copy(in->proposals[i].proposal);\n }\n out->proposal_is_cp=1;\n }\n)\n\nApop_settings_free(apop_mcmc, \n if (in->proposal_is_cp) {\n for (int i=0; i< in->block_count; i++)\n apop_model_free(in->proposals[i].proposal);\n free(in->proposals);\n }\n)\n\nstatic void setup_normal_proposals(apop_mcmc_proposal_s *s, int tsize, apop_mcmc_settings *settings){\n apop_model *mvn = apop_model_copy(apop_multivariate_normal);\n mvn->parameters = apop_data_alloc(tsize, tsize, tsize);\n gsl_vector_set_all(mvn->parameters->vector, 1);\n gsl_matrix_set_identity(mvn->parameters->matrix);\n s->proposal = mvn;\n s->step_fn = settings->base_step_fn;\n s->adapt_fn = settings->base_adapt_fn;\n}\n\nstatic void set_block_count_and_block_starts(apop_data *in, \n apop_mcmc_settings *s, size_t total_len){\n if (s->gibbs_chunks =='a') {\n s->block_count = 1;\n s->block_starts = calloc(2, sizeof(size_t));\n s->block_starts[1] = total_len;\n } else if (s->gibbs_chunks =='b') {\n s->block_count = 0;\n for (apop_data *d = in; d; d=d->more)\n s->block_count += !!d->vector + !!d->matrix + !!d->weights;\n\n s->block_starts = calloc(s->block_count+1, sizeof(size_t));\n int this=1, ctr=0;\n for (apop_data *d = in; d; d=d->more){\n #define markit(test, value) if (test) \\\n s->block_starts[this++] = ctr += value; \n\n markit(d->vector, d->vector->size);\n markit(d->matrix, d->matrix->size1*d->matrix->size2);\n markit(d->weights, d->weights->size);\n }\n } else { // item-by-item\n s->block_count = total_len;\n s->block_starts = calloc(total_len+1, sizeof(size_t));\n for (int i=1; iblock_starts[i] = i;\n }\n}\n\nstatic void one_step(apop_data *d, gsl_vector *draw, apop_model *m, apop_mcmc_settings *s, gsl_rng *rng, int *constraint_fails, apop_data *out, size_t block, int out_row){\n gsl_vector *clean_copy = apop_vector_copy(draw);\n newdraw:\n apop_draw(draw->data + s->block_starts[block], rng, s->proposals[block].proposal);\n apop_data_unpack(draw, m->parameters);\n if (m->constraint && m->constraint(d, m)){\n (*constraint_fails)++;\n goto newdraw;\n }\n double ll = apop_log_likelihood(d, m);\n\n Apop_notify(3, \"ll=%g for parameters:\\t\", ll);\n if (apop_opts.verbose >=3) apop_data_print(m->parameters, .output_pipe=apop_opts.log_file);\n\n Apop_stopif(gsl_isnan(ll) || !isfinite(ll), goto newdraw, \n 1, \"Trouble evaluating the m function at vector beginning with %g. \"\n \"Throwing it out and trying again.\\n\"\n , m->parameters->vector->data[0]);\n\n double ratio = ll - s->last_ll;\n if (ratio >= 0 || log(gsl_rng_uniform(rng)) < ratio){//success\n if (s->proposals[block].step_fn) \n s->proposals[block].step_fn(draw->data + s->block_starts[block], s->proposals+block, s);\n s->last_ll = ll;\n s->proposals[block].accept_count++;\n s->accept_count++;\n } else {\n s->proposals[block].reject_count++;\n s->reject_count++;\n Apop_notify(3, \"reject, with exp(ll_now-ll_proposal) = exp(%g-%g) = %g.\", ll, s->last_ll, exp(ratio));\n gsl_vector_memcpy(draw, clean_copy);\n apop_data_unpack(draw, m->parameters); //keep the last success in m->parameters.\n }\n if (out_row>=0) gsl_vector_memcpy(Apop_rv(out, out_row), draw);\n}\n\n\n/** The draw method for models estimated via \\ref apop_model_metropolis.\n\nThat method produces an \\ref apop_pmf, typically with a few thousand draws from the\nmodel in a batch. If you want to get a single next step from the Markov chain, use this.\n\nA Markov chain works by making a new draw and then accepting or rejecting the draw. If\nthe draw is rejected, the last value is reported as the next step in the chain. Users\nsometimes mitigate this repetition by making a batch of draws (say, ten at a time) and \nusing only the last.\n\nIf you run this without first running \\ref apop_model_metropolis, I will run it for\nyou, meaning that there will be an initial burn-in period before the first draw that\ncan be reported to you. That run is done using \\c model->data as input.\n\n\\param out An array of \\c doubles, which will hold the draw, in the style of \\ref apop_draw.\n\\param rng A \\c gsl_rng, already initialized, probably via \\ref apop_rng_alloc.\n\\param model A model which was probably already run through \\ref apop_model_metropolis.\n\\return On return, \\c out is filled with the next step in the Markov chain. The ->data element of the PMF model is extended to include the additional steps in the chain.\nIf a proposal failed the model constraints, then return 1; else return 0. See the notes in the documentation for \\ref apop_model_metropolis.\n\n \\li After pulling the attached settings group, the parent model is ignored. One expects\nthat \\c base_model in the mcmc settings group == the parent model.\n \\li If your settings break the model parameters into several chunks, this function\nreturns after stepping through all chunks.\n\\ingroup all_public\n*/\nint apop_model_metropolis_draw(double *out, gsl_rng* rng, apop_model *model){\n apop_mcmc_settings *s = apop_settings_get_group(model, apop_mcmc);\n if (!s || !s->pmf) {\n apop_model_metropolis(model->data, rng, model);\n s = apop_settings_get_group(model, apop_mcmc);\n }\n int constraint_fails = 0;\nOMP_critical (metro_draw)\n{\n apop_model *m = s->base_model;\n gsl_vector_view vv = gsl_vector_view_array(out, s->block_starts[s->block_count]);\n apop_data_pack(m->parameters, &(vv.vector));\n apop_data *earlier_draws = s->pmf->data;\n\n int block = 0, done = 0;\n while (!done){\n s->proposal_count++;\n earlier_draws->matrix = apop_matrix_realloc(earlier_draws->matrix, earlier_draws->matrix->size1+1, earlier_draws->matrix->size2);\n one_step(s->base_model->data, &(vv.vector), m, s, rng, &constraint_fails, \n earlier_draws, block, earlier_draws->matrix->size1-1);\n block = (block+1) % s->block_count;\n done = !block; //have looped back to the start.\n s->proposals[block].adapt_fn(s->proposals+block, s);\n }\n\n Apop_stopif(constraint_fails, , 2, \"%i proposals failed to meet your model's parameter constraints\", constraint_fails);\n}\n return !!constraint_fails;\n}\n\n\nvoid main_mcmc_loop(apop_data *d, apop_model *m, apop_data *out, gsl_vector *draw,\n apop_mcmc_settings *s, gsl_rng *rng, int *constraint_fails){\n\t\tdouble integerpart_periods_burnin = GSL_NAN; modf((double)(s->periods)*s->burnin,&integerpart_periods_burnin);\n s->accept_count = 0;\n int out_row = -lround(integerpart_periods_burnin);\n int block = 0;\n for (s->proposal_count=1; s->proposal_count< s->periods+1; s->proposal_count++, out_row++){\n one_step(d, draw, m, s, rng, constraint_fails, out, block, out_row);\n block = (block+1) % s->block_count;\n s->proposals[block].adapt_fn(s->proposals+block, s);\n //if (constraint_fails>10000) break;\n }\n}\n\n/** Use Metropolis-Hastings\nMarkov chain Monte Carlo to make draws from the given model.\n\nThe basic storyline is that draws are made from a proposal distribution, and the\nlikelihood of your model given your data and the drawn parameters evaluated. At each\nstep, a new set of proposal parameters are drawn, and if they are more likely\nthan the previous set the new proposal is accepted as the next step, else with probability (prob of new params)/(prob of old params),\nthey are accepted as the next step anyway. Otherwise the last accepted proposal is repeated.\n\nThe output is an \\ref apop_pmf model with a data set listing the draws that were\naccepted, including those repetitions. The output model is modified so that subsequent\ndraws are one more step from the Markov chain, via \\ref apop_model_metropolis_draw.\n\n\\param d The \\ref apop_data set used for evaluating the likelihood of a proposed parameter set.\n\n\\param rng A \\c gsl_rng, probably allocated via \\ref apop_rng_alloc. (Default: an RNG from \\ref apop_rng_get_thread)\n\n\\param m The \\ref apop_model from which parameters are being drawn. (No default; must not be \\c NULL)\n\n\\return A modified \\ref apop_pmf model representing the results of the search. It has\na specialized \\c draw method that returns another step from the Markov chain with each draw.\n\n\\exception out->error='c' Proposal was outside of a constraint; see below.\n\n\\li If a proposal fails to meet the \\c constraint element of the model you input, then\nthe proposal is thrown out and a new one selected. By the default proposal\ndistribution, this is not mathematically correct (it breaks detailed balance),\nand values near the constraint will be oversampled. The output model will have\noutmodel->error=='c'. It is up to you to decide whether the resulting\ndistribution is good enough for your purposes or whether to take the time to write a\ncustom proposal and step function to accommodate the constraint.\n\nAttach an \\ref apop_mcmc_settings group to your model to specify the proposal\ndistribution, burnin, and other details of the search. See the \\ref apop_mcmc_settings\ndocumentation for details.\n\n \\li The default proposal includes an adaptive step: you specify a target accept rate\n(default: .35), and if the accept rate is currently higher the variance of the proposals\nis widened to explore more of the space; if the accept rate is currently lower the\nvariance is narrowed to stay closer to the last accepted proposal. Technically, this\nbreaks ergodicity of the Markov chain, but the consensus seems to be that this is\nnot a serious problem. If it does concern you, you can set the \\c base_adapt_fn in the \\ref apop_mcmc_settings group to a do-nothing function, or one that damps its adaptation as \\f$n\\to\\infty\\f$.\n \\li If you have a univariate model, \\ref apop_arms_draw may be a suitable simpler alternative.\n \\li Note the \\c gibbs_chunks element of the \\ref apop_mcmc_settings group. If you set \\c\ngibbs_chunks='a', all parameters are drawn as a set, and accepted/rejected as a set. The\nvariances are adapted at an identical rate. If you set \\c gibbs_chunks='i',\nthen each scalar parameter is assigned its own proposal distribution, which is adapted\nat its own pace. With \\c gibbs_chunks='b' (the default), then each of the vector, matrix,\nand weights of your model's parameters are drawn/accepted/adapted as a block (and so\non to additional chunks if your model has ->more pages). This works well for\ncomplex models which naturally break down into subsets of parameters.\n \\li Each chunk counts as a step in the Markov chain. Therefore, if there are\nseveral chunks, you can expect chunks to repeat from step to step. If you want a\ndraw after cycling through all chunks, try using \\ref apop_model_metropolis_draw,\nwhich has that behavior.\n \\li If the likelihood model has \\c NULL parameters, I will allocate them. That\nmeans you can use one of the stock models that ship with Apophenia. If I need\nto run the model's prep routine to get the size of the parameters, then I will\nmake a copy of the likelihood model, run prep, and then allocate parameters\nfor that copy of a model.\n \\li On exit, the \\c parameters element of your likelihood model has the last accepted parameter proposal.\n \\li If you set apop_opts.verbose=2 or greater, I will report the accept\nrate of the M-H sampler. It is a common rule of thumb to select a proposal so that\nthis is between 20% and 50%. Set apop_opts.verbose=3 to see the stream\nof proposal points, their likelihoods, and the acceptance odds. You may want to\nset apop_opts.log_file=fopen(\"yourlog\", \"w\") first.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_model *apop_model_metropolis(apop_data *d, gsl_rng *rng, apop_model *m){\n apop_data *apop_varad_var(d, NULL);\n apop_model *apop_varad_var(m, NULL);\n Apop_stopif(!m, return NULL, 0, \"NULL model input.\");\n gsl_rng *apop_varad_var(rng, apop_rng_get_thread(-1));\nAPOP_VAR_END_HEAD\n apop_model *outp;\n OMP_critical(metropolis)\n {\n apop_mcmc_settings *s = apop_settings_get_group(m, apop_mcmc);\n if (!s)\n s = Apop_model_add_group(m, apop_mcmc);\n apop_prep(d, m); //typically a no-op\n s->last_ll = GSL_NEGINF;\n gsl_vector * drawv = apop_data_pack(m->parameters);\n const double double_periods = (double)(s->periods);\n Apop_stopif(s->burnin > 1, s->burnin/=double_periods,\n 1, \"Burn-in should be a fraction of the number of periods, \"\n \"not a whole number of periods. Rescaling to burnin=%g.\"\n , s->burnin/double_periods);\n\t\tdouble integerpart_periods_cburnin = GSL_NAN; modf(double_periods*(1.0-s->burnin),&integerpart_periods_cburnin);\n\t\tconst size_t data_size1 = llround(integerpart_periods_cburnin);\n apop_data *out = apop_data_alloc(data_size1, drawv->size);\n\n if (!s->proposals){\n set_block_count_and_block_starts(m->parameters, s, drawv->size);\n s->proposals = calloc(s->block_count, sizeof(apop_mcmc_proposal_s));\n s->proposal_is_cp = 1;\n for (int i=0; i< s->block_count; i++){\n apop_mcmc_proposal_s *p = s->proposals+i;\n setup_normal_proposals(p, s->block_starts[i+1]-s->block_starts[i], s);\n if (!p->proposal->parameters) {\n apop_prep(NULL, p->proposal+i);\n if(p->proposal->parameters->matrix) gsl_matrix_set_all(p->proposal->parameters->matrix, 1);\n if(p->proposal->parameters->vector) gsl_vector_set_all(p->proposal->parameters->vector, 1);\n }\n }\n }\n\n //if s->start_at =='p', we already have m->parameters in drawv.\n if (s->start_at == '1') gsl_vector_set_all(drawv, 1);\n int constraint_fails = 0;\n\n main_mcmc_loop(d, m, out, drawv, s, rng, &constraint_fails);\n\n Apop_notify(2, \"M-H sampling accept percent = %3.3f%%\", 100*(0.0+s->accept_count)/s->periods);\n Apop_stopif(constraint_fails, out->error='c', 2, \"%i proposals failed to meet your model's parameter constraints\", constraint_fails);\n\n out->weights = gsl_vector_alloc(s->periods*(1-s->burnin));\n gsl_vector_set_all(out->weights, 1);\n outp = apop_estimate(out, apop_pmf);\n s->pmf = outp;\n s->base_model = m;\n outp->draw = apop_model_metropolis_draw;\n apop_settings_copy_group(outp, m, \"apop_mcmc\");\n\n gsl_vector_free(drawv);\n }\n return outp;\n}\n" }, { "path": "apop_missing_data.m4.c", "content": "/** \\file apop_missing_data.c Some missing data handlers. */\n/* Copyright (c) 2007, 2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\n\n/** If there is an NaN anywhere in the row of data (including the matrix, the vector, the weights, and the text) then delete the row from the data set.\n\n\\li If every row has a NaN, then this returns \\c NULL.\n\\li If \\c apop_opts.nan_string is not \\c NULL, then I will make case-insensitive comparisons to the text elements to check for bad data as well.\n\\li If \\c inplace = 'y', then I'll free each element of the input data\n set and refill it with the pruned elements. I'll still take up (up to)\n twice the size of the data set in memory during the function. If\n every row has a NaN, then your \\c apop_data set will end up with\n \\c NULL vector, matrix, .... if \\c inplace = 'n', then the original data set is\n left where it was, though internal elements may be moved.\n\\li I only look at the first page of data (i.e. the \\c more element is ignored).\n\\li Listwise deletion is often not a statistically valid means of dealing with missing data.\n It is typically better to impute the data (preferably multiple times). See \\ref\n apop_ml_impute for a less-invalid means, or Tea\n for survey imputation for heavy-duty survey editing and imputation.\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\param d The data, with NaNs\n\\param inplace If \\c 'y', clear out the pointer-to-\\ref apop_data that\nyou sent in and refill with the pruned data. If \\c 'n', leave the\nset alone and return a new data set. Default=\\c 'n'.\n\\return A (potentially shorter) copy of the data set, without\nNaNs. If inplace=='y', a pointer to the input, which was shortened in place. If the entire data set is cleared out, then this will be \\c NULL.\n\\see apop_data_rm_rows\n*/\nAPOP_VAR_HEAD apop_data * apop_data_listwise_delete(apop_data *d, char inplace){\n apop_data * apop_varad_var(d, NULL);\n if (!d) return NULL;\n char apop_varad_var(inplace, 'n');\nAPOP_VAR_ENDHEAD\n Get_vmsizes(d) //defines firstcol, vsize, wsize, msize1, msize2.\n Apop_stopif(!msize1 && !vsize && !*d->textsize, return NULL, 0, \n \"You sent to apop_data_listwise_delete a data set with NULL matrix, NULL vector, and no text. \"\n \"Confused, it is returning NULL.\");\n //find out where the NaNs are\n int len = GSL_MAX(vsize ? vsize : msize1, d->textsize[0]); //still some size assumptions here.\n int not_empty = 0;\n int *marked = calloc(len, sizeof(int));\n for (int i=0; i< (vsize ? vsize: msize1); i++)\n for (int j=firstcol; j weights, i)))\n marked[i] = 1;\n if (d->textsize[0] && apop_opts.nan_string){\n for(int i=0; i< d->textsize[0]; i++)\n if (!marked[i])\n for(int j=0; j< d->textsize[1]; j++)\n if (!strcasecmp(apop_opts.nan_string, d->text[i][j])){\n marked[i] ++;\n break;\n }\n }\n\n //check that at least something isn't NULL.\n for (int i=0; i< len; i++)\n if (!marked[i]){\n not_empty ++;\n break;\n }\n if (!not_empty){\n free(marked);\n return NULL;\n }\n apop_data *out = (inplace=='y'|| inplace=='Y') ? d : apop_data_copy(d);\n apop_data_rm_rows(out, marked);\n free(marked);\n return out;\n}\n\n//ML imputation\n\n/** \\hideinitializer */\n#define Switch_back \\\n apop_data *real_data = ml_model->parameters; \\\n apop_model *actual_base = ml_model->more; \\\n actual_base->parameters = d; \n\nstatic void i_est(apop_data *d, apop_model *ml_model){\n Switch_back\n actual_base = apop_estimate(real_data, actual_base);\n}\n\nstatic long double i_ll(apop_data *d, apop_model *ml_model){\n Switch_back\n return apop_log_likelihood(real_data, actual_base);\n}\n\nstatic long double i_p(apop_data *d, apop_model *ml_model){\n Switch_back\n return apop_p(real_data, actual_base);\n}\n\n//doesn't actually move the parameters\nstatic long double i_constraint(apop_data *d, apop_model *ml_model){\n Switch_back\n if (!actual_base->constraint) return 0;\n apop_data *original_params = apop_data_copy(actual_base->parameters);\n long double out = actual_base->constraint(real_data, actual_base);\n if (out) apop_data_memcpy(actual_base->parameters, original_params);\n apop_data_free(original_params);\n return out;\n}\n\napop_model *apop_swap_model = &(apop_model){\"Model with data and params swapped\", .estimate=i_est, .p = i_p, .log_likelihood=i_ll, .constraint = i_constraint};\n\n/** Impute the most likely data points to replace NaNs in the data, and insert them into \nthe given data. That is, the data set is modified in place.\n\nHow it works: this uses the machinery for \\ref apop_model_fix_params. The only difference is \nthat this searches over the data space and takes the parameter space as fixed, while basic \nfix params model searches parameters and takes data as fixed. So this function just does the\nnecessary data-parameter switching to make that happen.\n\n\\param d The data set. It comes in with NaNs and leaves entirely filled in.\n\\param mvn A parametrized \\ref apop_model from which you expect the data was derived.\nif \\c NULL, then I'll use the Multivariate Normal that best fits the data after listwise deletion.\n\n\\return An estimated \\ref apop_model. Also, the data input will be filled in and ready to use.\n*/\napop_model * apop_ml_impute(apop_data *d, apop_model* mvn){\n if (!mvn){\n apop_data *list_d = apop_data_listwise_delete(d);\n Apop_stopif(!list_d, return NULL, 0, \"Listwise deletion returned no whole rows, \"\n \"so I couldn't fit a Multivariate Normal to your data. \"\n \"Please provide a pre-estimated initial model.\");\n mvn = apop_estimate(list_d, apop_multivariate_normal);\n apop_data_free(list_d);\n }\n apop_model *impute_me = apop_model_copy(apop_swap_model);\n impute_me->parameters = d;\n impute_me->more = mvn;\n apop_model *fixed = apop_model_fix_params(impute_me);\n Apop_model_add_group(fixed, apop_parts_wanted);\n apop_model *m = apop_estimate(mvn->parameters, fixed);\n apop_model_free(fixed);\n return m;\n}\n" }, { "path": "apop_mle.m4.c", "content": "/** \\file apop_mle.c */\n/*Copyright (c) 2006--2010 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n#include \"apop_internal.h\"\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\ntypedef long double (*apop_fn_with_params) (apop_data *, apop_model *);\ntypedef\tvoid (*apop_df_with_void)(const gsl_vector *beta, void *d, gsl_vector *gradient);\ntypedef\tvoid (*apop_fdf_with_void)(const gsl_vector *beta, void *d, double *f, gsl_vector *df);\n\n/** \\cond doxy_ignore */\ntypedef struct {\n\tgsl_vector\t*beta;\n\tint\t\t dimension;\n} grad_params;\n\ntypedef struct {\n apop_model *model;\n apop_data *data;\n apop_fn_with_params *f;\n grad_params *gp; //Used only by apop_internal_numerical_gradient.\n gsl_vector *beta, *starting_pt;\n int use_constraint;\n double best_ll;\n char want_cov, want_predicted, want_tests, want_info;\n jmp_buf bad_eval_jump;\n apop_data** path;\n} infostruct;\n/** \\endcond */ //End of Doxygen ignore.\n\nstatic apop_model * find_roots (infostruct p); //see end of file.\n\nm4_define(<|default_delta|>,1e-3) //as a macro, we can put it in documentation\n\n/* Generate support fns (esp. initializers) for apop_mle_settings and apop_parts_wanted structs. */\nApop_settings_copy(apop_parts_wanted, )\nApop_settings_free(apop_parts_wanted, )\nApop_settings_init(apop_parts_wanted,\n Apop_varad_set(covariance, 'n');\n Apop_varad_set(predicted, 'n');\n Apop_varad_set(tests, 'n');\n Apop_varad_set(info, 'n');\n)\n\nApop_settings_copy(apop_mle, )\nApop_settings_free(apop_mle, )\nApop_settings_init(apop_mle,\n Apop_varad_set(starting_pt, NULL);\n Apop_varad_set(tolerance, 1e-5);\n Apop_varad_set(max_iterations, 5000);\n Apop_varad_set(method, \"\");//default picked in apop_maximum_likelihood\n Apop_varad_set(verbose, 0);\n Apop_varad_set(step_size, 0.05);\n Apop_varad_set(delta, default_delta);\n Apop_varad_set(dim_cycle_tolerance, 0);\n//siman:\n //siman also uses step_size = 1.; \n Apop_varad_set(n_tries, 5); //The number of points to try for each step. \n Apop_varad_set(iters_fixed_T, 5); //The number of iterations at each temperature. \n Apop_varad_set(k, 1.0); //The maximum step size in the random walk. \n Apop_varad_set(t_initial, 50); //cooling schedule data\n Apop_varad_set(mu_t, 1.002); \n Apop_varad_set(t_min, 5.0e-1);\n Apop_varad_set(rng, NULL);\n)\n\n// MLE support functions\n//Including numerical differentiation and a couple of functions to\n//negate the likelihood fns without bothering the user.\n\nstatic void apop_annealing(infostruct*); //below.\n\nstatic double one_d(double b, void *in){\n infostruct *i = in;\n long double penalty = 0;\n gsl_vector_set(i->gp->beta, i->gp->dimension, b);\n apop_data_unpack(i->gp->beta, i->model->parameters);\n\tif (i->model->constraint)\n\t\tpenalty\t= i->model->constraint(i->data, i->model);\n\treturn (*(i->f))(i->data, i->model) + penalty;\n}\n\n//Numeric first and second derivatives.\n\n/* For each element of the parameter set, jiggle it to find its\n gradient. Return a vector as long as the parameter list. */\nstatic void apop_internal_numerical_gradient(apop_fn_with_params ll, \n infostruct* info, gsl_vector *out, double delta){\n double result, err;\n gsl_vector *beta = apop_data_pack(info->model->parameters);\n infostruct i = *info;\n i.f = ≪\n i.gp = &(grad_params){ .beta = gsl_vector_alloc(beta->size)};\n gsl_function F = { .function= one_d, \n .params\t= &i };\n\tfor (size_t j=0; j< beta->size; j++){\n\t\ti.gp->dimension = j;\n\t\tgsl_vector_memcpy(i.gp->beta, beta);\n\t\tgsl_deriv_central(&F, gsl_vector_get(beta,j), delta, &result, &err);\n\t\tgsl_vector_set(out, j, result);\n\t}\n gsl_vector_free(beta);\n}\n\n/**\nA wrapper around the GSL's one-dimensional \\c gsl_deriv_central to find a numeric differential for each dimension of the input \\ref apop_model's log likelihood (or \\c p if \\c log_likelihood is \\c NULL).\n\n\\param data The \\ref apop_data set to use for all evaluations.\n\\param model The \\ref apop_model, expressing the function whose derivative is sought. The gradient is taken via small changes along the model parameters.\n\\param delta The size of the differential. (default: default_delta, but see below)\n \n \\code\n gsl_vector *gradient = apop_numerical_gradient(data, your_parametrized_model);\n \\endcode\n\n\\li If you do not set \\ref delta as an input, I first look for an \\ref apop_mle_settings\n group attached to the input model, and check that for a \\c delta element. If that is\n also missing, use the default of default_delta.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD gsl_vector * apop_numerical_gradient(apop_data *data, apop_model *model, double delta){\n apop_data * apop_varad_var(data, NULL);\n apop_model * apop_varad_var(model, NULL);\n Nullcheck(model, NULL) Nullcheck_p(model, NULL)\n double apop_varad_var(delta, 0);\n if (!delta){\n apop_mle_settings *mp = apop_settings_get_group(model, apop_mle);\n delta = mp ? mp->delta : default_delta;\n }\nAPOP_VAR_ENDHEAD\n Get_vmsizes(model->parameters); //tsize\n apop_fn_with_params ll = model->log_likelihood ? model->log_likelihood : model->p;\n Apop_stopif(!ll, return NULL, 0, \"Input model has neither p nor log_likelihood method. Returning NULL.\");\n gsl_vector *out = gsl_vector_calloc(tsize);\n infostruct i = (infostruct) {.model = model, .data = data};\n apop_internal_numerical_gradient(ll, &i, out, delta);\n return out;\n}\n\n/** \\cond doxy_ignore */\ntypedef struct {\n apop_model *base_model;\n int *current_index;\n} apop_model_for_infomatrix_struct;\n/** \\endcond */\n\nstatic long double apop_fn_for_infomatrix(apop_data *d, apop_model *m){\n static threadlocal gsl_vector *v = NULL;\n apop_model_for_infomatrix_struct *settings = m->more;\n apop_model *mm = settings->base_model;\n apop_score_type ms = apop_score_vtable_get(mm);\n if (ms){\n Get_vmsizes(mm->parameters); //tsize\n if (!v || v->size != tsize){\n if (v) gsl_vector_free(v);\n v = gsl_vector_alloc(tsize);\n }\n ms(d, v, mm);\n return gsl_vector_get(v, *settings->current_index);\n } //else:\n gsl_vector *vv = apop_numerical_gradient(d, mm);\n double out = gsl_vector_get(vv, *settings->current_index);\n gsl_vector_free(vv);\n return out;\n}\n\napop_model *apop_model_for_infomatrix = &(apop_model){\"Ad hoc model for working out the information matrix.\", \n .log_likelihood = apop_fn_for_infomatrix};\n\n/** Numerically estimate the matrix of second derivatives of the parameter values, via\na series of re-evaluations at small differential steps. [Therefore, it may be expensive\nto do this for a very computationally-intensive model.]\n\n\\param data The \\ref apop_data at which the model was estimated (default: \\c NULL)\n\\param model The \\ref apop_model, with parameters already estimated (no default, must not be \\c NULL)\n\\param delta the step size for the differentials. (default: default_delta, but see below)\n\\return The matrix of estimated second derivatives at the given data and parameter values.\n \n\\li If you do not set \\ref delta as an input, I first look for an \\ref apop_mle_settings group attached to the input model, and check that for a \\c delta element. If that is also missing, use the default of default_delta.\n\\li This function uses the \\ref designated syntax for inputs.\n */\nAPOP_VAR_HEAD apop_data * apop_model_hessian(apop_data * data, apop_model *model, double delta){\n apop_data * apop_varad_var(data, NULL);\n apop_model * apop_varad_var(model, NULL);\n Nullcheck(model, NULL)\n double apop_varad_var(delta, 0);\n if (!delta){\n apop_mle_settings *mp = apop_settings_get_group(model, apop_mle);\n delta = mp ? mp->delta : default_delta;\n }\nAPOP_VAR_ENDHEAD\n int k;\n Get_vmsizes(model->parameters) //tsize\n size_t betasize = tsize;\n apop_data *out = apop_data_calloc(0, betasize, betasize);\n gsl_vector *dscore = gsl_vector_alloc(betasize);\n apop_model_for_infomatrix_struct ms = { .base_model = model, .current_index = &k, };\n apop_model *m = apop_model_copy(apop_model_for_infomatrix);\n m->parameters = model->parameters;\n m->more = &ms;\n if (apop_settings_get_group(model, apop_mle))\n apop_settings_copy_group(m, model, \"apop_mle\");\n for (k=0; k< betasize; k++){\n dscore = apop_numerical_gradient(data, m, delta);\n //We get two estimates of the (k,j)th element, which are often very close,\n //and take the mean.\n for (size_t j=0; j< betasize; j++){\n *gsl_matrix_ptr(out->matrix, k, j) += gsl_vector_get(dscore, j)/2;\n *gsl_matrix_ptr(out->matrix, j, k) += gsl_vector_get(dscore, j)/2;\n }\n gsl_vector_free(dscore);\n }\n if (model->parameters->names->row){\n apop_name_stack(out->names, model->parameters->names, 'r');\n apop_name_stack(out->names, model->parameters->names, 'c', 'r');\n }\n return out;\n}\n\n/** Produce the covariance matrix for the parameters of an estimated model via the derivative of the score function at the parameter. I.e., I find the second derivative via \\ref apop_model_hessian , and take the negation of the inverse.\n\nI follow Efron and Hinkley in using the estimated information matrix---the value of the information matrix at the estimated value of the score---not the expected information matrix that is the integral over all possible data. See Pawitan 2001 (who cribbed a little off of Efron and Hinkley) or Klemens 2008 (who directly cribbed off of both) for further details. \n\n\\param data The data by which your model was estimated\n\\param model A model whose parameters have been estimated.\n\\param delta The differential by which to step for sampling changes. (default: default_delta, but see below)\n\\return A covariance matrix for the data. Also, if the data does not have a\n \"\" page, I'll set it to the result as well [i.e., I won't overwrite an\n existing covariance page]. \n\n\\li If you do not set \\ref delta as an input, I first look for an \\ref apop_mle_settings group attached to the input model, and check that for a \\c delta element. If that is also missing, use the default of default_delta.\n\\li This function uses the \\ref designated syntax for inputs.\n */\nAPOP_VAR_HEAD apop_data * apop_model_numerical_covariance(apop_data * data, apop_model *model, double delta){\n apop_data * apop_varad_var(data, NULL);\n apop_model * apop_varad_var(model, NULL);\n Nullcheck(model, NULL)\n double apop_varad_var(delta, 0);\n if (!delta){\n apop_mle_settings *mp = apop_settings_get_group(model, apop_mle);\n delta = mp ? mp->delta : default_delta;\n }\nAPOP_VAR_ENDHEAD\n apop_data *hessian = apop_model_hessian(data, model, delta);\n if (apop_opts.verbose > 1){\n printf(\"The estimated Hessian:\\n\");\n apop_data_show(hessian);\n }\n apop_data *out = apop_data_alloc();\n out->matrix = apop_matrix_inverse(hessian->matrix);\n gsl_matrix_scale(out->matrix, -1);\n if (hessian->names->row){\n apop_name_stack(out->names, hessian->names, 'r');\n apop_name_stack(out->names, hessian->names, 'c');\n }\n apop_data_free(hessian);\n if (!apop_data_get_page(model->parameters, \"\"))\n apop_data_add_page(model->parameters, out, \"\");\n return out;\n}\n\n///On to the interfaces between the models and the methods\n\nstatic void tracepath(const gsl_vector *beta, double value, apop_data **path){\n size_t msize1 = (*path && (*path)->matrix) ? (*path)->matrix->size1: 0;\n if (!*path) {\n *path = apop_data_alloc();\n (*path)->names->title = strdup(\"Path of ML search\");\n apop_name_add((*path)->names, \"f(x)\", 'v');\n apop_name_add((*path)->names, \"x\", 'm');\n }\n (*path)->matrix = apop_matrix_realloc((*path)->matrix, msize1+1, beta->size);\n gsl_vector_memcpy(Apop_rv(*path, msize1), beta);\n\n (*path)->vector = apop_vector_realloc((*path)->vector, msize1+1);\n gsl_vector_set((*path)->vector, msize1, value);\n}\n\n/* Every actual evaluation of the function go through the negshell and dnegshell fns,\n because there are several things that have to be done beyond just getting\n model.log_likelihood:\n\n--Negate, because statisticians and social scientists like to maximize; physicists like to minimize.\n--Work out if the model provides log_likelihood or p.\n--Call \\ref trace_path if needed.\n--Go from a single vector to a full apop_data set and back (via apop_data_pack/unpack)\n--Check the derivative function if available.\n--Check constraints.\n*/\n\nstatic double negshell (const gsl_vector *beta, void * in){\n infostruct *i = in;\n double penalty = 0,\n out = 0; \n long double (*f)(apop_data *, apop_model *);\n f = i->model->log_likelihood? i->model->log_likelihood : i->model->p;\n Apop_stopif(!f, longjmp(i->bad_eval_jump, -1),\n 0, \"The model you sent to the MLE function has neither log_likelihood element nor p element.\");\n apop_data_unpack(beta, i->model->parameters);\n\tif (i->use_constraint && i->model->constraint)\n\t\tpenalty\t= i->model->constraint(i->data, i->model);\n if (penalty) apop_data_pack(i->model->parameters, (gsl_vector*) beta);\n double f_val = f(i->data, i->model);\n out = penalty - f_val; //negative llikelihood\n Apop_stopif(gsl_isnan(out), longjmp(i->bad_eval_jump, -1),\n 0, \"I got a NaN in evaluating the objective function.%s\", \n !i->model->constraint ? \" Maybe add a constraint to your model?\" : \"\");\n if (i->path) tracepath(i->model->parameters->vector, -out, i->path);\n if (i->want_info =='y'){\n //I report the log likelihood under the assumption that the final param set \n //matches the best ll evaluated.\n long double this_ll = i->model->log_likelihood? -out : log(-out); //negative negative llikelihood.\n\n if(gsl_isnan(this_ll)){\n Apop_stopif(!i->model->log_likelihood && penalty > f_val, i->want_info='n',\n 1, \"Model's p=%g, penalty=%g, for a negative adjusted p=%g. \"\n \"Continuing, but can not report covariance or \"\n \"log likelihood-based statistics.\", f_val, penalty, f_val-penalty);\n Apop_stopif(1, apop_data_show(i->model->parameters); i->want_info='n',\n 1, \"NaN resulted from the following value tried by the maximum likelihood system.\");\n }\n i->best_ll = GSL_MAX(i->best_ll, this_ll);\n }\n return out;\n}\n\nstatic int dnegshell (const gsl_vector *beta, void * in, gsl_vector * g){\n/* The derivative-calculating routine.\nIf the constraint binds\n then: take the numerical derivative of negshell, which will be the\n numerical derivative of the penalty.\n else: just find dlog_likelihood. If the model doesn't have a\n dlog likelihood or the user asked to ignore it, then the main\n maximum likelihood fn replaced model.score with\n apop_numerical_gradient anyway.\nFinally, reverse the sign, since the GSL is trying to minimize instead of maximize.\n*/\n infostruct *i = in;\n apop_mle_settings *mp = apop_settings_get_group(i->model, apop_mle);\n apop_data_unpack(beta, i->model->parameters);\n /* In all cases, negshell gets called first, so the constraint is already\n checked and beta nudged accordingly.\n if(i->model->constraint && i->model->constraint(i->data, i->model))\n apop_data_pack(i->model->parameters, (gsl_vector *) beta); */\n apop_score_type ms = apop_score_vtable_get(i->model);\n if (ms) ms(i->data, g, i->model);\n else {\n apop_fn_with_params ll = i->model->log_likelihood ? i->model->log_likelihood : i->model->p;\n apop_internal_numerical_gradient(ll, i, g, mp->delta);\n }\n if (mp->path) negshell (beta, in);\n gsl_vector_scale(g, -1);\n return GSL_SUCCESS;\n}\n\n//This is just to satisfy the GSL's format.\nstatic void fdf_shell(const gsl_vector *beta, void *i, double *f, gsl_vector *df){\n *f\t= negshell(beta, i);\n dnegshell(beta, i, df);\n}\n\nstatic int ctrl_c;\nstatic void mle_sigint(int){ ctrl_c ++; }\n\nstatic int setup_starting_point(apop_mle_settings *mp, gsl_vector *x){\n Apop_stopif(!x, return -1, 0, \"The vector I'm trying to optimize over is NULL.\");\n\tif (!mp->starting_pt) gsl_vector_set_all (x, 1);\n\telse for (int i=0; i< x->size; i++)\n x->data[i] = mp->starting_pt[i];\n return 0;\n}\n\nvoid add_info_criteria(apop_data *d, apop_model *m, apop_model *est, double ll, int param_ct){\n //Did the sending function save last value of f()?\n if (!ll) ll = apop_log_likelihood(d, m);\n\n if (!est->info) est->info = apop_data_alloc();\n apop_data_add_named_elmt(est->info, \"log likelihood\", ll);\n double AIC = 2*param_ct - 2 *ll;\n apop_data_add_named_elmt(est->info, \"AIC\", AIC);\n if (d){//some models have NULL data.\n int n;\n {Get_vmsizes(d); n = maxsize;}\n apop_data_add_named_elmt(est->info, \"AIC_c\", AIC + 2*param_ct *(param_ct + 1.0)/(n - param_ct - 1.0));\n Get_vmsizes(d); //vsize, msize1, tsize\n apop_data_add_named_elmt(est->info, \"BIC\", param_ct * log(n) - 2 *ll);\n }\n}\n\nstatic void auxinfo(apop_data *params, infostruct *i, int status, double ll){\n apop_model *est = i->model; //just an alias.\n /* This catches too many near-misses\n if(est->constraint)\n apop_assert(!est->constraint(i->data, est), \"the maximum likelihood search ended \"\n \"at a point that doesn't satisfy the model's constraints.\");*/\n if (i->want_cov=='y' && est->parameters){\n apop_model_numerical_covariance(i->data, est, Apop_settings_get(est,apop_mle,delta));\n if (i->want_tests=='y')\n apop_estimate_parameter_tests (est);\n }\n if (!est->info) est->info = apop_data_alloc();\n apop_data_add_named_elmt(est->info, \"status\", status);\n if (i->want_info=='y') add_info_criteria(i->data, i->model, est, ll, i->beta->size);\n}\n\nstatic void apop_maximum_likelihood_w_d(apop_data * data, infostruct *i){\n/* The maximum likelihood calculations, given a derivative of the log likelihood.\n\nIf no derivative exists, will calculate a numerical gradient.\n\nInside the infostruct, you'll find these elements:\n\n\\param data\tthe data matrix\n\\param\tdist\tthe \\ref apop_model object: probit, zipf, &c.\n\\param\tstarting_pt\tan array of doubles suggesting a starting point. If NULL, use a vector whose elements are all 0.1 (zero has too many pathological cases).\n\\param step_size\tthe initial step size.\n\\param tolerance\tthe precision the minimizer uses. Only vaguely related to the precision of the actual var.\n\\return\tan \\ref apop_model with the parameter estimates, &c. If returned_estimate->status == 0, then optimum parameters were found; if status != 0, then there were problems.\n*/\n gsl_multimin_fdfminimizer *s;\n apop_model\t\t*est = i->model; //just an alias.\n apop_mle_settings *mp = apop_settings_get_group(est, apop_mle);\n int iter \t= 0, \n\t status = 0,\n\t apopstatus = 0,\n\t betasize= i->beta->size;\n if (!strcasecmp(mp->method, \"BFGS cg\"))\n\t s = gsl_multimin_fdfminimizer_alloc(gsl_multimin_fdfminimizer_vector_bfgs2, betasize);\n else if (!strcasecmp(mp->method, \"PR cg\"))\n\t s = gsl_multimin_fdfminimizer_alloc(gsl_multimin_fdfminimizer_conjugate_pr, betasize);\n else //Default: \"FR CG\" conjugate gradient (Fletcher-Reeves)\n\t s = gsl_multimin_fdfminimizer_alloc(gsl_multimin_fdfminimizer_conjugate_fr, betasize);\n gsl_multimin_function_fdf minme = {\n .f\t\t= negshell,\n .df\t = (apop_df_with_void) dnegshell,\n .fdf\t= (apop_fdf_with_void) fdf_shell,\n .n\t\t= betasize,\n .params\t= i};\n ctrl_c = 0;\n if (setjmp(i->bad_eval_jump))\n Apop_stopif(1, return, 0, \"Failure evaluating likelihood at the starting point. Add a starting point?\");\n\tgsl_multimin_fdfminimizer_set (s, &minme, i->beta, mp->step_size, mp->tolerance);\n signal(SIGINT, mle_sigint);\n do { \t\n iter++;\n if (setjmp(i->bad_eval_jump)) {\n apopstatus = -1;\n break;\n }\n status \t= gsl_multimin_fdfminimizer_iterate(s);\n if(status && status!=GSL_CONTINUE) break; //commented out error msg because too many GSL_ENOPROG false positives.\n //Apop_stopif(status && status!=GSL_CONTINUE, break, 0, \"GSL error: %s\", gsl_strerror(status));\n status = gsl_multimin_test_gradient(s->gradient, mp->tolerance);\n if(status && status!=GSL_CONTINUE) break; //commented out error msg because too many GSL_ENOPROG false positives.\n //Apop_stopif(status && status!=GSL_CONTINUE, break, 0, \"GSL error: %s\", gsl_strerror(status));\n if (mp->verbose)\n printf (\"%5i %.5f f()=%10.5f gradient=%.3f\\n\", iter, gsl_vector_get (s->x, 0), s->f, gsl_vector_get(s->gradient,0));\n Apop_stopif(status == GSL_SUCCESS, apopstatus=0, 2, \"Optimum found.\");\n } while (status == GSL_CONTINUE && iter < mp->max_iterations && !ctrl_c);\n signal(SIGINT, NULL);\n\tApop_stopif(iter==mp->max_iterations, apopstatus = -1, 1, \"Max iterations reached, implying that I did not find an optimum.\");\n\t//Clean up, copy results to output estimate.\n apop_data_unpack(s->x, est->parameters);\n\tgsl_multimin_fdfminimizer_free(s);\n gsl_vector_free(i->beta);\n auxinfo(est->parameters, i, apopstatus, i->best_ll);\n}\n\n/* See apop_maximum_likelihood_w_d for notes. */\nstatic void apop_maximum_likelihood_no_d(apop_data * data, infostruct * i){\n apop_model *est = i->model;\n apop_mle_settings *mp = apop_settings_get_group(est, apop_mle);\n int status=0,\n apopstatus = 0,\n iter = 0,\n betasize= i->beta->size;\n gsl_multimin_fminimizer *s;\n gsl_vector *ss;\n double size;\n s = gsl_multimin_fminimizer_alloc(gsl_multimin_fminimizer_nmsimplex, betasize);\n ss = gsl_vector_alloc(betasize);\n ctrl_c =\n apopstatus = 0; //assume failure until we score a success.\n gsl_vector_set_all (ss, mp->step_size);\n gsl_multimin_function minme = {.f = negshell, .n= betasize, .params = i};\n if (setjmp(i->bad_eval_jump))\n Apop_stopif(1, return, 0, \"Failure evaluating likelihood at the starting point. Add a starting point?\");\n gsl_multimin_fminimizer_set (s, &minme, i->beta, ss);\n //i->beta = s->x;\n signal(SIGINT, mle_sigint);\n do { \n iter++;\n if (setjmp(i->bad_eval_jump)) {\n apopstatus = -1;\n break;\n }\n status = gsl_multimin_fminimizer_iterate(s);\n if (status) break; \n size = gsl_multimin_fminimizer_size(s);\n status = gsl_multimin_test_size (size, mp->tolerance); \n if(mp->verbose){\n printf (\"%5d \", iter);\n for (size_t j = 0; j < betasize; j++) \n printf (\"%8.3e \", gsl_vector_get (s->x, j)); \n printf (\"f()=%7.3f size=%.3f\\n\", s->fval, size);\n if (status == GSL_SUCCESS) {\n printf (\"Optimum found at:\\n\");\n printf (\"%5d \", iter);\n for (size_t j = 0; j < betasize; j++)\n printf (\"%8.3e \", gsl_vector_get (s->x, j)); \n printf (\"f()=%7.3f size=%.3f\\n\", s->fval, size);\n }\n }\n } while (status == GSL_CONTINUE && iter < mp->max_iterations && !ctrl_c);\n signal(SIGINT, NULL);\n\tApop_stopif(iter == mp->max_iterations && mp->verbose, /*continue*/, \n 1, \"Optimization reached maximum number of iterations.\");\n if (status == GSL_SUCCESS) apopstatus = 0;\n apop_data_unpack(s->x, est->parameters);\n\tgsl_multimin_fminimizer_free(s);\n auxinfo(est->parameters, i, apopstatus, i->best_ll);\n}\n\n/*There is a basically standard location for the log likelihood. Search there, and if you don't\nfind it, then recalculate it.*/\nstatic double get_ll(apop_data *d, apop_model *est){\n int index = est->info ? apop_name_find(est->info->names, \"log likelihood\", 'r') : -2;\n if (index>-2) return apop_data_get(est->info, index);\n //last resort: recalculate\n return apop_log_likelihood(d, est);\n}\n\nstatic void dim_cycle(apop_data *d, apop_model *est, infostruct info){\n double last_ll, this_ll = GSL_NEGINF;\n int iteration = 0;\n apop_mle_settings *mp = Apop_settings_get_group(est, apop_mle);\n double tol = mp->dim_cycle_tolerance;\n int betasize = info.beta->size;\n Apop_settings_set(est, apop_mle, dim_cycle_tolerance, 0);//so sub-estimations won't use this function.\n gsl_vector *paramv = apop_data_pack(est->parameters);\n apop_model *full_est = NULL; //an alias\n do {\n if (mp->verbose){\n if (!(iteration++))\n printf(\"Cycling toward an optimum. Listing (dim):log likelihood.\\n\");\n printf(\"Iteration %i:\\n\", iteration);\n }\n last_ll = this_ll;\n for (int i=0; i< betasize; i++){\n gsl_vector_set(info.beta, i, GSL_NAN);\n apop_data_unpack(info.beta, est->parameters);\n apop_model *m_onedim = apop_model_fix_params(est);\n apop_prep(d, m_onedim);\n apop_maximum_likelihood(d, m_onedim);\n gsl_vector_set(info.beta, i, m_onedim->parameters->vector->data[0]);\n full_est = apop_model_fix_params_get_base(m_onedim);//points to est, but filled.\n this_ll = get_ll(d, full_est);//only used on the last iteration.\n if (mp->verbose) printf(\"(%i):%g\\t\", i, this_ll), fflush(NULL);\n apop_model_free(m_onedim);\n }\n if (mp->verbose) printf(\"\\n\");\n apop_data_pack(full_est->parameters, paramv);\n Apop_settings_add(est, apop_mle, starting_pt, paramv->data);\n } while (fabs(this_ll - last_ll) > tol);\n Apop_settings_set(est, apop_mle, dim_cycle_tolerance, tol);\n gsl_vector_free(paramv);\n}\n\nvoid get_desires(apop_model *m, infostruct *info){\n apop_parts_wanted_settings *want = apop_settings_get_group(m, apop_parts_wanted);\n\n info->want_tests = (want && want->tests =='y') ? 'y' : 'n';\n info->want_cov = (info->want_tests=='y' || (want && want->covariance =='y'))\n ? 'y' : 'n';\n info->want_info = want ? (want->info =='y' ? 'y' : 'n') : 'y';\n\n //doesn't do anything at the moment.\n info->want_predicted = (want && want->predicted =='y') ? 'y' : 'n';\n}\n\nint check_method (char *m){\n#define Onecheck(str) if (!strcasecmp(m, #str)) return 0;\nif(!m || strlen(m)==0) return 0;\nOnecheck(NM simplex)\nOnecheck(FR cg)\nOnecheck(BFGS cg)\nOnecheck(PR cg)\nOnecheck(Annealing)\nOnecheck(Newton)\nOnecheck(Newton hybrid)\nOnecheck(Newton hybrid no scale)\nreturn 1;\n}\n\n/** Find the likelihood-maximizing parameters of a model given data.\n\n\\li I assume that \\ref apop_prep has been called on your model. The easiest way to guarantee this is to use \\ref apop_estimate, which calls this function if the input model has no \\c estimate method.\n\n\\li All of the settings are specified by adding a\n \\ref apop_mle_settings struct to your model, so see the many notes there. Notably,\n the default method is the Fletcher-Reeves conjugate gradient method, and if your model\n does not have a dlog likelihood function, then a numeric gradient will be calculated\n via \\ref apop_numerical_gradient. Add an \\ref apop_mle_settings group to your model\n to set tuning parameters or select other methods, including the Nelder-Mead simplex,\n simulated annealing, and root-finding.\n\n\\param data\t An \\ref apop_data set.\n\n\\param\tdist\tThe \\ref apop_model object: \\ref apop_gamma, \\ref apop_probit, \\ref apop_zipf, &c. You can add\n an \\c apop_mle_settings struct to it (Apop_model_add_group(your_model, apop_mle,\n .verbose=1, .method=\"PR cg\", and_so_on)).\n\n\\return\tNone, but the input model is modified to include the parameter estimates, &c. \n\n\\li There is auxiliary info in the ->info element of the post-estimation struct. Get elements via, e.g.:\n\\code\napop_model *est = apop_estimate(your_data, apop_probit);\n\n\nint status = apop_data_get(est->info, .rowname=\"status\");\nif (status)\n //trouble\nelse\n //optimum found\n apop_data_print(est->parameters); //Here are the estimated parameters\n\\endcode\n\n\\li During the search for an optimum, ctrl-C (SIGINT) will halt the search, and the function will return whatever parameters the search was on at the time.\n*/\nvoid apop_maximum_likelihood(apop_data * data, apop_model *dist){\n apop_mle_settings *mp = apop_settings_get_group(dist, apop_mle);\n if (!mp) mp = Apop_model_add_group(dist, apop_mle);\n apop_score_type ms = apop_score_vtable_get(dist);\n Apop_stopif(check_method(mp->method), return, 0, \"You set the method='%s', \"\n \"which is not on my list of allowable methods. See the apop_mle_settings \"\n \"documentation for the list of options\", mp->method);\n if (!mp->method || !strlen(mp->method)) mp->method = ms ? \"FR cg\" : \"NM simplex\";\n\n Apop_stopif(!dist->parameters, dist->error='p'; return, 0, \"Not enough information to allocate parameters over which to optimize. If this was not called from apop_estimate, did you call apop_prep first?\");\n infostruct info = {.data = data,\n .use_constraint = 1,\n .path = mp->path,\n .model = dist};\n get_desires(dist, &info);\n info.beta = apop_data_pack(dist->parameters);\n if (setup_starting_point(mp, info.beta)) return;\n info.model->data = data;\n if (mp->dim_cycle_tolerance) dim_cycle(data, dist, info);\n else if (!strcasecmp(mp->method, \"annealing\")) apop_annealing(&info); //below.\n else if (!strcasecmp(mp->method, \"NM simplex\")) apop_maximum_likelihood_no_d(data, &info);\n else if (!strcasecmp(mp->method, \"Newton\") ||\n !strcasecmp(mp->method, \"Newton hybrid\")||\n !strcasecmp(mp->method, \"Newton hybrid no scale\")) find_roots (info);\n else /* Conjugate Gradient*/ apop_maximum_likelihood_w_d(data, &info);\n}\n\n/** Maximum likelihod searches are not guaranteed to find a global optimum, and it can be\ndifficult to tune a search such that it covers a wide space, but also accurately hones in\non the optimum. In both cases, one could restart the search using a different starting\npoint or different parameters.\n\nThe simplest use of this function is to restart a model at the latest parameter estimates.\n\n \\code\napop_model *m = apop_estimate(data, model_using_an_MLE_search);\nfor (int i=0; i< 10; i++)\n m = apop_estimate_restart(m);\napop_data_show(m);\n \\endcode\n\nBy adding a line to reduce the tolerance each round [e.g., Apop_settings_set(m, apop_mle, tolerance, pow(10,-i))], you can start broad and hone in on a precise optimum.\n \nYou may have a new estimation method, such as first doing a coarse simulated annealing search, then a fine conjugate gradient search. When reading this example, recall that the form for adding a new settings group differs from the form for modifying existing settings:\n \\code\nApop_model_add_settings(your_base_model, apop_mle, .method=APOP_SIMAN);\napop_model *m = apop_estimate(data, your_base_model);\nApop_settings_set(m, apop_mle, method, APOP_CG_PR);\nm = apop_estimate_restart(m);\napop_data_show(m);\n \\endcode\n\nOnly one estimate is returned, either the one you sent in or a new\none. The loser (which may be the one you sent in) is freed, to prevent memory leaks.\n\n\\param e An \\ref apop_model that is the output from a prior MLE estimation. (No default, must not be \\c NULL.)\n\\param copy Another not-yet-parametrized model that will be re-estimated with (1) the same data and (2) a starting_pt as per the next setting (probably\n to the parameters of e). If this is NULL, then copy e. (Default = \\c NULL)\n\\param starting_pt \"ep\"=last estimate of the first model (i.e., its current parameter estimates)
\n\"es\"= starting point originally used by the first model
\n\"np\"=current parameters of the new (second) model
\n\"ns\"=starting point specified by the new model's MLE settings. (default = \"ep\")\n\\param boundary I test whether the starting point you give me has magintude greater\n than this bound, so I can warn you if there's divergence in your sequence of\n re-estimations. (default: 1e8)\n\n\\return If the new estimated parameters include any NaNs/Infs, then\n the old estimate is returned (even if the old estimate included\n NaNs/Infs). Otherwise, the estimate with the largest log likelihood\n is returned.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/ \nAPOP_VAR_HEAD apop_model * apop_estimate_restart (apop_model *e, apop_model *copy, char * starting_pt, double boundary){\n apop_model * apop_varad_var(e, NULL);\n Nullcheck_m(e, NULL);\n apop_model * apop_varad_var(copy, NULL);\n char * apop_varad_var(starting_pt, \"ep\");\n double apop_varad_var(boundary, 1e8);\nAPOP_VAR_ENDHEAD\n gsl_vector *v = NULL;\n if (!copy) copy = apop_model_copy(e);\n apop_mle_settings* prm0 = apop_settings_get_group(e, apop_mle);\n apop_mle_settings* prm = apop_settings_get_group(copy, apop_mle);\n //copy off the old params; modify the starting pt, method, and scale\n if (!strcmp(starting_pt, \"es\"))\n v = apop_array_to_vector(prm0->starting_pt);\n else if (!strcmp(starting_pt, \"ns\")){\n int size =sizeof(prm->starting_pt)/sizeof(double);\n v = apop_array_to_vector(prm->starting_pt, size);\n prm0->starting_pt\t= malloc(sizeof(double)*size);\n memcpy(prm0->starting_pt, prm->starting_pt, sizeof(double)*size);\n }\n else if (!strcmp(starting_pt, \"np\")){\n v = apop_data_pack(copy->parameters); \n prm->starting_pt = malloc(sizeof(double)*v->size);\n memcpy(prm->starting_pt, v->data, sizeof(double)*v->size);\n }\n else if (e->parameters){//\"ep\" or default.\n v = apop_data_pack(e->parameters); \n prm->starting_pt = malloc(sizeof(double)*v->size);\n memcpy(prm->starting_pt, v->data, sizeof(double)*v->size);\n }\n Apop_stopif(!apop_vector_bounded(v, boundary), return e, \n 0, \"Your model has diverged (element(s) > %g);\"\n \" returning your original model without restarting.\", boundary);\n gsl_vector_free(v);\n \n apop_model *newcopy = apop_estimate(e->data, copy);\n apop_model_free(copy);\n //Now check whether the new output is better than the old\n if (apop_vector_bounded(newcopy->parameters->vector, boundary) \n && get_ll(e->data, newcopy) > get_ll(e->data, e)){\n apop_model_free(e);\n return newcopy;\n } //else:\n apop_model_free(newcopy);\n return e;\n}\n\n// Simulated Annealing.\n\nstatic double annealing_energy(void *in) {\n infostruct *i = in;\n return negshell(i->beta, i);\n}\n\nstatic double annealing_distance(void *xin, void *yin) {\n/** We use the Manhattan metric to correspond to the annealing_step fn below. */\n gsl_vector *from = apop_vector_copy(((infostruct*)xin)->beta);\n gsl_vector *to = apop_vector_copy(((infostruct*)yin)->beta);\n gsl_vector_div(from, ((infostruct*)xin)->starting_pt);\n gsl_vector_div(to, ((infostruct*)xin)->starting_pt);//starting pts are the same.\n return apop_vector_distance(from, to, .metric='m');\n}\n\nstatic void annealing_check_constraint(infostruct *i){\n apop_data_unpack(i->beta, i->model->parameters);\n if (i->model->constraint && i->model->constraint(i->data, i->model))\n apop_data_pack(i->model->parameters, i->beta);\n}\n\nstatic void annealing_step(const gsl_rng * r, void *in, double step_size){\n/** The algorithm: \n --randomly pick dimension\n --shift by some amount of remaining step size\n --repeat for all dims\nThis will give a move \\f$\\leq\\f$ step_size on the Manhattan metric.\n*/\n infostruct *i = in;\n int sign;\n double amt, scale;\n double cutpoints[i->beta->size+1];\n cutpoints[0] = 0;\n cutpoints[i->beta->size] = 1;\n for (size_t j=1; j< i->beta->size; j++)\n cutpoints[j] = gsl_rng_uniform(r);\n\n for (size_t j=0; j< i->beta->size; j++){\n sign = (gsl_rng_uniform(r) > 0.5) ? 1 : -1;\n scale = gsl_vector_get(i->starting_pt, j);\n amt = cutpoints[j+1]- cutpoints[j];\n *gsl_vector_ptr(i->beta, j) += amt * sign * scale * step_size;\n }\n annealing_check_constraint(i);\n}\n\nstatic void annealing_print(void *xp) {\n apop_vector_show(((infostruct*)xp)->beta);\n}\n\nstatic void annealing_print2(void *xp) { return; }\n\nstatic void annealing_memcpy(void *xp, void *yp){\n infostruct *yi = yp;\n infostruct *xi = xp;\n *yi = *xi;\n yi->beta = apop_vector_copy(xi->beta);\n}\n\nstatic void *annealing_copy(void *xp){\n infostruct *out = malloc(sizeof(infostruct));\n annealing_memcpy(xp, out);\n return out;\n}\n\nstatic void annealing_free(void *xp){\n gsl_vector_free(((infostruct*)xp)->beta);\n free(xp);\n}\n\n//I abuse the starting point element to hold the list of scaling factors. They can't be zero.\nstatic double set_start(double in){ return in ? in : 1; }\n\njmp_buf anneal_jump;\nstatic void anneal_sigint(int){ longjmp(anneal_jump,1); }\n\nstatic void apop_annealing(infostruct *i){\n apop_model *ep = i->model;\n apop_mle_settings *mp = apop_settings_get_group(ep, apop_mle);\n Apop_stopif(!mp, ep->error='l'; return, 0, \"The model you sent to the MLE function has neither log_likelihood element nor p element.\");\n gsl_siman_params_t simparams = (gsl_siman_params_t) {\n .n_tries = mp->n_tries, \n .iters_fixed_T = mp->iters_fixed_T,\n .step_size = mp->step_size,\n .k = mp->k,\n .t_initial = mp->t_initial,\n .mu_t = mp->mu_t,\n .t_min = mp->t_min};\n gsl_rng *r = mp->rng ? mp->rng : apop_rng_get_thread();\n //these two are done at apop_maximum_likelihood:\n //i->beta = apop_data_pack(ep->parameters);\n //setup_starting_point(mp, i->beta);\n int betasize = i->beta->size;\n int apopstatus = -1;\n i->starting_pt = apop_vector_map(i->beta, set_start);\n i->use_constraint = 0; //negshell doesn't check it; annealing_step does.\n gsl_siman_print_t printing_fn = NULL;\n if (mp && mp->verbose>1) printing_fn = annealing_print;\n else if (mp && mp->verbose) printing_fn = annealing_print2;\n annealing_check_constraint(i); //shift starting point if needed.\n if (setjmp(i->bad_eval_jump)) {\n apopstatus = -1;\n goto done;\n }\n if (!setjmp(anneal_jump)){\n signal(SIGINT, anneal_sigint);\n gsl_siman_solve(r, // const gsl_rng * r\n i, // void * x0_p\n annealing_energy, // gsl_siman_Efunc_t Ef\n annealing_step, // gsl_siman_step_t take_step\n annealing_distance, // gsl_siman_metric_t distance\n printing_fn, // gsl_siman_print_t print_position\n annealing_memcpy, // gsl_siman_copy_t copyfunc\n annealing_copy, // gsl_siman_copy_construct_t copy_constructor\n annealing_free, // gsl_siman_destroy_t destructor\n betasize, // size_t element_size\n simparams); // gsl_siman_params_t params\n }\n signal(SIGINT, NULL);\n apop_data_unpack(i->beta, i->model->parameters); \n apop_estimate_parameter_tests(i->model);\n apopstatus = 0;\ndone:\n if (mp->rng) r = NULL;\n auxinfo(i->model->parameters, i, apopstatus, i->best_ll);\n}\n\n/* This function calls the various GSL root-finding algorithms to find the zero of the score.\n Cut/pasted/modified from the GSL documentation. */\nstatic apop_model * find_roots (infostruct p) {\n const gsl_multiroot_fsolver_type *T;\n gsl_multiroot_fsolver *s;\n apop_model *dist = p.model;\n apop_mle_settings *mlep = apop_settings_get_group(dist, apop_mle);\n int status=0, betasize = p.beta->size,\n apopstatus = -1; //assume failure until we score a success.\n size_t iter = 0;\n gsl_multiroot_function f = {dnegshell, betasize, &p};\n T = !strcasecmp(mlep->method, \"Newton\") ? gsl_multiroot_fsolver_dnewton\n : !strcasecmp(mlep->method, \"Newton hybrid no scale\") ? gsl_multiroot_fsolver_hybrids\n : gsl_multiroot_fsolver_hybrid;\n s = gsl_multiroot_fsolver_alloc (T, betasize);\n gsl_multiroot_fsolver_set (s, &f, p.beta);\n do {\n iter++;\n if (setjmp(p.bad_eval_jump)) break;\n status = gsl_multiroot_fsolver_iterate (s);\n if (!mlep || mlep->verbose)\n printf (\"iter = %3zu x = % .3f f(x) = % .3e\\n\", iter, gsl_vector_get (s->x, 0), gsl_vector_get (s->f, 0));\n if (status) /* check if solver is stuck */\n break;\n status = gsl_multiroot_test_residual (s->f, mlep->tolerance);\n } while (status == GSL_CONTINUE && iter < mlep->max_iterations);\n if (GSL_SUCCESS) apopstatus = 0;\n Apop_notify(2, \"status = %s\\n\", gsl_strerror(status));\n apop_data_unpack(s->x, dist->parameters);\n gsl_multiroot_fsolver_free (s);\n gsl_vector_free (p.beta);\n auxinfo(dist->parameters, &p, apopstatus, 0); //root-finders don't store best val.\n return dist;\n}\n" }, { "path": "apop_model.m4.c", "content": "/** \\file apop_model.c\t sets up the estimate structure which outputs from the various regressions and MLEs.*/\n/* Copyright (c) 2006--2011 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#define Declare_type_checking_fns\n#include \"apop_internal.h\"\n\n/** Set up the \\c parameters and \\c info elements of the \\c apop_model: \n\nAt close, the input model has parameters of the correct size.\n\n\\li This is the default action for \\ref apop_prep, and many models with a custom prep routine\ncall \\ref apop_model_clear at the end. Also, \\ref apop_estimate calls this function internally, which means that you robably never have to call this function directly.\n\\li If the model has already been prepped, this function should be a no-op.\n\n\\param data If your params vary with the size of the data set, then the function needs a data set to calibrate against. Otherwise, it's OK to set this to \\c NULL.\n\\param model The model whose output elements will be modified.\n\\return A pointer to the same model, should you need it.\n\\exception outmodel->error=='d' dimension error.\n*/\napop_model * apop_model_clear(apop_data * data, apop_model *model){\n Get_vmsizes(data)\n int width = msize2 ? msize2 : -firstcol;//use the vector only if there's no matrix.\n Apop_stopif(model->dsize==-1 && !width, model->error='d', 0, \"The model's dsize==-1, meaning size=data width, but the input data has NULL vector and matrix.\");\n Apop_stopif(model->vsize==-1 && !width, model->error='d', 0, \"The model's vsize==-1, meaning size=data width, but the input data has NULL vector and matrix.\");\n Apop_stopif(model->msize1==-1 && !width, model->error='d', 0, \"The model's msize1==-1, meaning size=data width, but the input data has NULL vector and matrix.\");\n Apop_stopif(model->msize2==-1 && !width, model->error='d', 0, \"The model's msize2==-1, meaning size=data width, but the input data has NULL vector and matrix.\");\n\n model->dsize = (model->dsize == -1 ? width : model->dsize);\n vsize = model->vsize == -1 ? width : model->vsize;\n msize1 = model->msize1 == -1 ? width : model->msize1 ;\n msize2 = model->msize2 == -1 ? width : model->msize2 ;\n if (!model->parameters && (vsize || msize1*msize2)) \n model->parameters = apop_data_alloc(vsize, msize1, msize2);\n if (!model->info) model->info = apop_data_alloc();\n if (model->info->names->title && !strlen(model->info->names->title))\n free(model->info->names->title);\n Asprintf(&model->info->names->title, \"\");\n if (!model->data) model->data = data;\n\treturn model;\n}\n\n/** Free an \\ref apop_model structure.\n\n \\li The \\c parameters and \\c settings are freed. These are the elements that are\ncopied by \\c apop_model_copy.\n \\li The \\c data element is not freed, because the odds are you still need it.\n \\li If free_me->more_size is positive, the function runs\nfree(free_me->more). But it has no idea what the \\c more element contains;\nif it points to other structures (like an \\ref apop_data set), you need to free them\nbefore calling this function.\n \\li If \\c free_me is \\c NULL, this does nothing.\n\n\\param free_me A pointer to the model to be freed.\n*/\nvoid apop_model_free (apop_model * free_me){\n if (!free_me) return;\n apop_data_free(free_me->parameters);\n if (free_me->settings){\n int i=0;\n while (free_me->settings[i].name[0]){\n if (free_me->settings[i].free)\n ((void (*)(void*))(free_me->settings[i].free))(free_me->settings[i].setting_group);\n i++;\n }\n free(free_me->settings);\n }\n if (free_me->more_size)\n free(free_me->more);\n if (free_me->info)\n apop_data_free(free_me->info);\n\tfree(free_me);\n}\n\n/** Print the results of an estimation for a human to look over.\n\n\\param model The model whose information should be displayed (No default. If \\c NULL, print NULL)\n\\param output_pipe The output stream. Default: \\c stdout. If you'd like something else, use \\c fopen. E.g.:\n\\code\nFILE *out =fopen(\"outfile.txt\", \"w\"); //or \"a\" to append.\napop_model_print(the_model, out);\nfclose(out); //optional in many cases.\n\\endcode\n\n\\li The default prints the name, parameters, info, &c. but I check a vtable for\nalternate methods you define; see \\ref vtables for details. The typedef new functions\nmust conform to and the hash used for lookups are:\n\n\\code\ntypedef void (*apop_model_print_type)(apop_model *params, FILE *out);\n#define apop_model_print_hash(m1) ((m1)->log_likelihood ? (size_t)(m1)->log_likelihood : \\\n (m1)->p ? (size_t)(m1)->p*33 : \\\n (m1)->estimate ? (size_t)(m1)->estimate*33*33 : \\\n (m1)->draw ? (size_t)(m1)->draw*33*27 : \\\n (m1)->cdf ? (size_t)(m1)->cdf*27*27 : 27)\n\\endcode\n\nWhen building a special print method, all output should \\c fprintf to the input \\c FILE* handle. \n Apophenia's output routines also accept a file handle; e.g., if the file handle is\n named \\c out, then if the \\c thismodel print method uses \\c apop_data_print to\n print the parameters, it must do so via a form like apop_data_print(thismodel->parameters,\n .output_pipe=ap).\n\nYour \\c print method can use both by masking itself for a few lines:\n \\code\nvoid print_method(apop_model *in, FILE* ap){\n void *temp = in->estimate;\n in->estimate = NULL;\n apop_model_print(in, ap);\n in->estimate = temp;\n\n printf(\"Additional info:\\n\");\n ...\n}\n \\endcode\n\n\\li Print methods are intended for human consumption and are subject to change.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD void apop_model_print (apop_model * model, FILE *output_pipe){\n FILE * apop_varad_var(output_pipe, stdout);\n apop_model* apop_varad_var(model, NULL);\n if (!model) {fprintf(output_pipe, \"NULL\\n\"); return;}\nAPOP_VAR_ENDHEAD\n apop_model_print_type mpf = apop_model_print_vtable_get(model);\n if (mpf){\n mpf(model, output_pipe);\n return;\n }\n if (strlen(model->name)) fprintf (output_pipe, \"%s\", model->name);\n fprintf(output_pipe, \"\\n\\n\");\n\tif (model->parameters) apop_data_print(model->parameters, .output_pipe=output_pipe);\n Get_vmsizes(model->info); //maxsize\n if (model->info && maxsize) apop_data_print(model->info, .output_pipe=output_pipe);\n}\n\n/* Alias for \\ref apop_model_print. Use that one. */\nvoid apop_model_show (apop_model * print_me){\n apop_model_print(print_me, NULL);\n}\n\n/** Outputs a copy of the \\ref apop_model input.\n\n\\param in The model to be copied\n\\return A copy of the original. Includes copies\nof all settings groups, and the \\c parameters (if not \\c NULL, copied via \\ref\napop_data_copy).\n\n\\li If in.more_size > 0 I memcpy the \\c more pointer from the original data set.\n\\li The data set at \\c in->data is not copied, but is also pointed to.\n\n\\exception out->error=='a' Allocation error. In extreme cases, where there aren't even a few hundred bytes available, I will return \\c NULL.\n\\exception out->error=='s' Error copying settings groups.\n\\exception out->error=='p' Error copying parameters or info page; the given \\ref apop_data struct may be \\c NULL or may have its own ->error element.\n*/\napop_model * apop_model_copy(apop_model *in){\n Apop_stopif(!in, return NULL, 1, \"Copying a NULL input; returning NULL.\");\n apop_model * out = malloc(sizeof(apop_model));\n Apop_stopif(!out, return NULL, 0, \"Serious allocation error; returning NULL.\");\n memcpy(out, in, sizeof(apop_model));\n if (in->more_size){\n out->more = malloc(in->more_size);\n Apop_stopif(!out->more, out->error='a'; return out, 0, \"Allocation error setting up the ->more pointer.\");\n memcpy(out->more, in->more, in->more_size);\n }\n int i=0; \n out->settings = NULL;\n if (in->settings)\n do \n apop_settings_copy_group(out, in, in->settings[i].name);\n while (strlen(in->settings[i++].name));\n out->parameters = apop_data_copy(in->parameters);\n Apop_stopif(in->parameters && (!out->parameters || out->parameters->error), \n out->error='p'; return out, 0, \"Error copying the model parameters.\");\n out->info = apop_data_copy(in->info);\n Apop_stopif(in->info && (!out->info || out->info->error), \n out->error='p'; return out, 0, \"Error copying the info segment.\");\n return out;\n}\n\n/** \\def apop_model_set_parameters(in, ...)\nTake in an unparameterized \\c apop_model and return a new \\c apop_model with the given parameters. \nFor example, if you need a N(0,1) quickly:\n\\code\napop_model *std_normal = apop_model_set_parameters(apop_normal, 0, 1);\n\\endcode\n\nThis doesn't take in data, so it won't work with models that take the number of\nparameters from the data, and it will only set the vector of the model's parameter \\ref\napop_data set. This is most standard models. If you have a situation where these options\nare out, you could\n\\li manually set Set \\c .vsize and/or \\c .msize1 and \\c .msize2 first, then call this function, or\n\\li prep the model via something like apop_model *new = apop_model_copy(in);\napop_prep(your_data, new); (because \\ref apop_prep is required to correctly\nallocate \\c new->parameters to conform to your data).\n\n\\param in An unparameterized model, like \\ref apop_normal or \\ref apop_poisson.\n\\param ... The list of parameters.\n\\return A copy of the input model, with parameters set.\n\\exception out->error=='d' dimension error: you gave me a model with an indeterminate\nnumber of parameters. See notes above.\nSet \\c .vsize or \\c .msize1 and \\c .msize2 first, then call this function, or use\napop_model *new = apop_model_copy(in); apop_prep(your_data, new); and then\ncall this .\n\\see apop_data_fill\n\\hideinitializer \n*/\napop_model *apop_model_set_parameters_base(apop_model *in, double ap[]){\n apop_model *out = apop_model_copy(in);\n apop_prep(NULL, out);\n Apop_stopif((in->vsize == -1) || (in->msize1 == -1) || (in->msize2 == -1), out->error='d', \n 0, \"This function only works with models whose number of params does not \"\n \"depend on data size. You'll have to use apop_model *new = apop_model_copy(in); \"\n \" apop_model_clear(your_data, in); and then set in->parameters using your data.\");\n apop_data_fill_base(out->parameters, ap);\n return out; \n}\n\n/** Estimate the parameters of a model given data.\n\nThis function copies the input model, preps it (see \\ref apop_prep), and calls \\c\nm.estimate(d, m) (which users are encouraged to never call directly). If your model\nhas no \\c estimate method, then call \\c apop_maximum_likelihood(d, m), with the default\nMLE settings.\n\n\\param d The data\n\\param m The model\n\\return A pointer to an output model, which typically matches the input model but has its \\c parameters element filled in.\n*/\napop_model *apop_estimate(apop_data *d, apop_model *m){\n apop_model *out = apop_model_copy(m);\n apop_prep(d, out);\n if (out->estimate) out->estimate(d, out); \n else apop_maximum_likelihood(d, out);\n return out;\n}\n\n/** Find the probability of a data/parametrized model pair.\n\n\\param d The data\n\\param m The parametrized model, which must have either a \\c log_likelihood or a \\c p method.\n*/\ndouble apop_p(apop_data *d, apop_model *m){\n Nullcheck_m(m, GSL_NAN);\n if (m->p)\n return m->p(d, m);\n else if (m->log_likelihood)\n return exp(m->log_likelihood(d, m));\n Apop_stopif(0, , 0, \"You asked for the probability of a model that has neither p nor log_likelihood methods.\");\n return GSL_NAN;\n}\n\n/** Find the log likelihood of a data/parametrized model pair.\n\n\\param d The data\n\\param m The parametrized model, which must have either a \\c log_likelihood or a \\c p method.\n*/\ndouble apop_log_likelihood(apop_data *d, apop_model *m){\n Nullcheck_m(m, GSL_NAN); //Nullcheck_p(m); //Too many models don't use the params.\n if (m->log_likelihood)\n return m->log_likelihood(d, m);\n else if (m->p)\n return log(m->p(d, m));\n Apop_stopif(0, , 0, \"You asked for the log likelihood of a model that has neither p nor log_likelihood methods.\");\n return GSL_NAN;\n}\n\n/** Find the vector of first derivatives (aka the gradient) of the log likelihood of a data/parametrized model pair.\n\nOn input, the model \\c m must already be sufficiently prepped\nthat the log likelihood can be evaluated; see \\ref psubsection for details.\n\nOn output, the \\c gsl_vector input to the function will be filled with the gradients\n(or NaNs on errors). If the model parameters have a more complex shape\nthan a simple vector, then the vector will be in \\c apop_data_pack order; use \\c\napop_data_unpack to reformat to the preferred shape.\n\n\\param d The \\ref apop_data set at which the score is being evaluated.\n\\param out The score to be returned. I expect you to have allocated this already.\n\\param m The parametrized model, which must have either a \\c log_likelihood or a \\c p method.\n\n\\li The default is to use \\ref apop_numerical_gradient, but special-case calculations\nfor certain models are held in a vtable; see \\ref vtables for details. The typedef\nnew functions must conform to and the hash used for lookups are:\n\n\\code\ntypedef void (*apop_score_type)(apop_data *d, gsl_vector *gradient, apop_model *m);\n#define apop_score_hash(m1) ((size_t)((m1).log_likelihood ? (m1).log_likelihood : (m1).p))\n\\endcode\n*/\nvoid apop_score(apop_data *d, gsl_vector *out, apop_model *m){\n Nullcheck_m(m, );\n Apop_stopif(!out, return, 0, \"out vector is NULL. It must be pre-allocated to the correct size. E.g., gsl_vector *out = gsl_vector_alloc(m->vsize + m->size1*m->size2))).\");\n apop_score_type ms = apop_score_vtable_get(m);\n if (ms){\n ms(d, out, m);\n return;\n }\n gsl_vector * numeric_default = apop_numerical_gradient(d, m);\n gsl_vector_memcpy(out, numeric_default);\n gsl_vector_free(numeric_default);\n}\n\nApop_settings_init(apop_pm,\n //defaults include base=NULL, index=0, own_rng=0\n Apop_varad_set(rng, NULL);\n Apop_varad_set(draws, 1e4);\n)\n\nApop_settings_copy(apop_pm,)\n\nApop_settings_free(apop_pm, )\n\nvoid distract_doxygen(){/*Doxygen gets thrown by the settings macros. This decoy function is a workaround. */}\n\n/** Get a model describing the distribution of the given parameter estimates.\n\nFor many models, the parameter estimates are well-known, such as the\n\\f$t\\f$-distribution of the parameters for OLS.\n\nFor models where the distribution of \\f$\\hat{p}\\f$ is not known, if you give me data, I\nwill return an \\ref apop_normal or \\ref apop_multivariate_normal model, using the parameter estimates as mean and \\ref apop_bootstrap_cov for the variances.\n\nIf you don't give me data, then I will assume that this is a stochastic model where \nre-running the model will produce different parameter estimates each time. In this case, I will\nrun the model 1e4 times and return a \\ref apop_pmf model with the resulting parameter\ndistributions.\n\nBefore calling this, I expect that you have already run \\ref apop_estimate to produce \\f$\\hat{p}\\f$.\n\nThe \\ref apop_pm_settings structure dictates details of how the model is generated.\nFor example, if you want only the distribution of the third parameter, and you know the\ndistribution will be a PMF generated via random draws, then set settings and call the\nmodel via:\n\\code\n apop_model_group_add(your_model, apop_pm, .index =3, .draws=3e5);\n apop_model *dist = apop_parameter_model(your_data, your_model);\n\\endcode\n\nSome useful parts of \\ref apop_pm_settings:\n\\li \\c index gives the position of the parameter (in \\ref apop_data_pack order)\nin which you are interested. Thus, if this is zero or more, then you will get a\nunivariate output distribution describing a single parameter. If index == -1,\nthen I will give you the multivariate distribution across all parameters. The default\nis zero (i.e. the univariate distribution of the zeroth parameter).\n\\li \\c draws If there is no closed-form solution and bootstrap is inappropriate, then\nthe last resort is a large numbr of random draws of the model, summarized into a PMF. Default: 1,000 draws.\n\\li \\c rng If the method requires random draws, then use this. If you provide \\c NULL and one is needed, I provide one for you via \\ref apop_rng_get_thread.\n\nThe default is via resampling as above, but special-case calculations for certain models are held in a vtable; see \\ref vtables for details. The typedef new functions must conform to and the hash used for lookups are:\n\n\\code\ntypedef apop_model* (*apop_parameter_model_type)(apop_data *, apop_model *);\n#define apop_parameter_model_hash(m1) ((size_t)((m1).log_likelihood ? (m1).log_likelihood : (m1).p)*33 + (m1).estimate ? (size_t)(m1).estimate: 27)\n\\endcode\n*/ \napop_model *apop_parameter_model(apop_data *d, apop_model *m){\n apop_pm_settings *settings = apop_settings_get_group(m, apop_pm);\n if (!settings)\n settings = Apop_settings_add_group(m, apop_pm, .base= m);\n apop_parameter_model_type pm = apop_parameter_model_vtable_get(m);\n if (pm) return pm(d, m);\n else if (d){\n Get_vmsizes(m->parameters);//vsize, msize1, msize2\n apop_model *out = apop_model_copy(apop_multivariate_normal);\n out->msize1 = out->vsize = out->msize2 = out->dsize = vsize+msize1+msize2;\n out->parameters = apop_bootstrap_cov(d, m, settings->rng, settings->draws);\n out->parameters->vector = apop_data_pack(m->parameters);\n if (settings->index == -1)\n return out;\n else {\n apop_model *out2 = apop_model_set_parameters(apop_normal, \n apop_data_get(out->parameters, settings->index, -1), //mean\n apop_data_get(out->parameters, settings->index, settings->index)//var\n );\n apop_model_free(out);\n return out2;\n }\n } //else\n Get_vmsizes(m->parameters);//vsize, msize1, msize2\n apop_data *param_draws = apop_data_alloc(0, settings->draws, vsize+msize1+msize2);\n for (int i=0; i < settings->draws; i++){\n apop_model *mm = apop_estimate (NULL, m);//If you're here, d==NULL.\n apop_data_pack(mm->parameters, Apop_rv(param_draws, i));\n apop_model_free(mm);\n }\n if (settings->index == -1)\n return apop_estimate(param_draws, apop_pmf);\n else {\n apop_data *param_draws1 = apop_data_alloc(settings->draws, 0,0);\n gsl_vector *the_draws = Apop_cv(param_draws, settings->index);\n gsl_vector_memcpy(param_draws1->vector, the_draws);\n apop_data_free(param_draws);\n return apop_estimate(param_draws1, apop_pmf);\n }\n}\n\nextern apop_model *apop_swap_model; //apop_missing_data.c\nint apop_model_metropolis_draw(double *out, gsl_rng* rng, apop_model *params);//apop_update.c\n\n/** Draw from a model. \n\n\\param out An already-allocated array of doubles to be filled by the draw method. It must have size m->dsize.\n\\param r A \\c gsl_rng, probably allocated via \\ref apop_rng_alloc. Optional; if \\c NULL, then I will call \\ref apop_rng_get_thread for an RNG.\n\\param m The model from which to make draws.\n\n\\li If the model has its own \\c draw method, then this function will call it.\n\\li Else, if the model is univariate, use \\ref apop_arms_draw to generate random draws.\n\\li Else, if the model is multivariate, use \\ref apop_model_metropolis to generate random draws.\n\\li This makes a single draw of the given size. See \\ref apop_model_draws to fill a matrix with draws.\n\n\\return Zero on success; nozero on failure. out[0] is probably \\c NAN on failure.\n*/\nint apop_draw(double *out, gsl_rng *r, apop_model *m){\n if (!r) r = apop_rng_get_thread(-1);\n if (m->draw)\n return m->draw(out, r, m); \n else if (m->dsize == 1)\n return apop_arms_draw(out, r, m);\n //Else, MCMC, possibly setting it up first.\n //generate a model with data/params reversed\n //estimate mcmc. Swapped model will be stored as settings->base_model.\n OMP_critical (apop_draw)\n if (!Apop_settings_get_group(m, apop_mcmc)){\n apop_model *swapped = apop_model_copy(apop_swap_model);\n swapped->more = m;\n swapped->msize1 = 1;\n swapped->msize2 = m->dsize;\n swapped->data = m->parameters;\n Apop_settings_add_group(swapped, apop_mcmc, .burnin=0.999, .periods=1000);\n apop_model *est = apop_model_metropolis(m->parameters, r, swapped); //leak.\n m->draw = apop_model_metropolis_draw;\n apop_settings_copy_group(m, est, \"apop_mcmc\");\n }\n return apop_draw(out, r, m);\n}\n\n/** Allocate and initialize the \\c parameters, \\c info, and other requisite parts of a \\ref apop_model.\n\nSome models have associated prep routines that also attach settings groups to the model, and set up additional special-case functions in vtables.\n\n\\li The input model is modified in place.\n\\li If called repeatedly, subsequent calls to \\ref apop_prep are no-ops. Thus, a model\n can not be re-prepped using a new data set or other conditions.\n\\li The default prep is to simply call \\ref apop_model_clear. If the\n input \\ref apop_model has a prep method, then that gets called instead.\n*/\nvoid apop_prep(apop_data *d, apop_model *m){\n if (m->prep) m->prep(d, m);\n else apop_model_clear(d, m);\n}\n\nstatic double disnan(double in) {return gsl_isnan(in);}\n\n/** A prediction supplies E(a missing value | original data, already-estimated parameters, and other supplied data elements ).\n\nFor a regression, one would first estimate the parameters of the model, then supply a row of predictors X. The value of the dependent variable \\f$y\\f$ is unknown, so the system would predict that value.\n\nFor a univariate model (i.e. a model in one-dimensional data space), there is only one variable to omit and fill in, so the prediction problem reduces to the expected value: E(a missing value | original data, already-estimated parameters). [In some models, this may not be the expected value, but is a best value for the missing item using some other meaning of `best'.]\n\nIn other cases, prediction is the missing data problem: for three-dimensional data,\nyou may supply the input (34, \\c NaN, 12), and the parameterized model provides the\nmost likely value of the middle parameter given the parameters and known data.\n\n\\li If you give me a \\c NULL data set, I will assume you want all values filled in, for most models with the expected value.\n\n\\li If you give me data with \\c NaNs, I will take those as the points to\nbe predicted given the provided data.\n\nIf the model has no \\c predict method, the default is to use the \\ref apop_ml_impute function to do the work. That function does a maximum-likelihood search for the best parameters.\n\n\\return If you gave me a non-\\c NULL data set, I will return that, with the \\c NaNs filled in. If \\c NULL input, I will allocate an \\ref apop_data set and fill it with the expected values.\n\nThere may be a second page (i.e., a \\ref apop_data set attached to the ->more pointer of the main) listing confidence and standard error information. See your specific model documentation for details.\n\n\\li Special-case calculations for certain models are held in a vtable; see \\ref vtables for details. The typedef new functions must conform to and the hash used for lookups are:\n\n\\code\ntypedef apop_data * (*apop_predict_type)(apop_data *d, apop_model *params);\n#define apop_predict_hash(m1) ((size_t)((m1).log_likelihood ? (m1).log_likelihood : (m1).p)*33 + (m1).estimate ? (size_t)(m1).estimate: 27)\n\\endcode\n*/\napop_data *apop_predict(apop_data *d, apop_model *m){\n apop_data *prediction = NULL;\n apop_data *out = d ? d : apop_data_alloc(0, 1, m->dsize);\n if (!d) gsl_matrix_set_all(out->matrix, GSL_NAN);\n apop_predict_type mp = apop_predict_vtable_get(m);\n if (mp) prediction = mp(out, m);\n if (prediction) return prediction;\n if (!apop_map_sum(out, disnan)) return out;\n //default:\n apop_model *f = apop_ml_impute(out, m);\n apop_model_free(f);\n return out;\n}\n\n/* Are all the elements of v less than or equal to the corresponding elements of the reference vector? */\nstatic int lte(gsl_vector *v, gsl_vector *ref){\n for (int i=0; i< v->size; i++) \n if(v->data[i] > gsl_vector_get(ref, i))\n return 0;\n return 1;\n}\n\n/** Input a one-row data point/vector and a model; returns the area of the model's PDF beneath the given point.\n\nBy default, make random draws from the PDF and return the percentage of those\ndraws beneath or equal to the given point. Many models have closed-form solutions that\nmake no use of random draws. \n\nSee also \\ref apop_cdf_settings, which is the structure used to store draws already\nmade (which means the second, third, ... calls to this function will take much less\ntime than the first), the \\c gsl_rng, and the number of draws to be made. These are\nhandled without your involvement, but if you would like to change the number of draws\nfrom the default, add this group before calling \\ref apop_cdf :\n\n\\code\nApop_model_add_group(your_model, apop_cdf, .draws=1e5, .rng=my_rng);\ndouble cdf_value = apop_cdf(your_data_point, your_model);\n\\endcode\n\n\\li Only the first row of the input \\ref apop_data set is used. Note that if you need to view row 20 of a data set as a one-row data set, use \\ref Apop_r.\n\nHere are many examples using common, mostly symmetric distributions.\n\n\\include some_cdfs.c\n*/\ndouble apop_cdf(apop_data *d, apop_model *m){\n if (m->cdf) return m->cdf(d, m);\n apop_cdf_settings *cs = Apop_settings_get_group(m, apop_cdf);\n if (!cs) cs = Apop_model_add_group(m, apop_cdf);\n long int tally = 0; \n \n gsl_vector *ref = apop_data_pack(Apop_r(d, 0));\n if (!cs->draws_made){\n if (m->dsize == -1) apop_prep(d, m);\n Apop_stopif(m->dsize==0, return GSL_NAN, 0, \"I need to make random draws from your model, but it has dsize==0. Returning NaN\");\n cs->draws_made = gsl_matrix_alloc(cs->draws, m->dsize);\n for (int i=0; i< cs->draws; i++)\n apop_draw((Apop_mrv(cs->draws_made, i))->data, cs->rng, m);\n }\n for (int i=0; i< cs->draws_made->size1; i++)\n tally += lte(Apop_mrv(cs->draws_made, i), ref);\n gsl_vector_free(ref);\n return tally/(double)cs->draws_made->size1;\n}\n\nApop_settings_init(apop_cdf,\n Apop_varad_set(draws, 1e4);\n Apop_varad_set(rng, NULL);\n out->draws_refcount = malloc(sizeof(int));\n *out->draws_refcount = 1;\n)\n\nApop_settings_free(apop_cdf,\n if (in->draws_made && !--*in->draws_refcount)\n gsl_matrix_free(in->draws_made);\n)\n\nApop_settings_copy(apop_cdf,\n ++*out->draws_refcount;\n)\n" }, { "path": "apop_name.m4.c", "content": "/** \\file apop_name.c */\n/* Copyright (c) 2006--2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n#include \n\n/** Allocates a name structure\n\\return\tAn allocated, empty name structure. In the very unlikely event that \\c malloc fails, return \\c NULL.\n\nBecause \\ref apop_data_alloc uses this to set up its output, you will rarely if ever\nneed to call this function explicitly. You may want to use it if wrapping a \\c gsl_matrix into an \\ref apop_data set. For example, to put a title on a vector:\n\n\\code\napop_data *d = &(apop_data){.vector=your_vector, .names=apop_name_alloc()};\napop_name_add(d->names, \"A column of numbers\", 'v');\napop_data_print(d);\n\n...\napop_name_free(d->names); //but d itself is auto-allocated; no need to free it.\n\\endcode\n*/\napop_name * apop_name_alloc(void){\n apop_name * init_me = malloc(sizeof(apop_name));\n Apop_stopif(!init_me, return NULL, 0, \"malloc failed. Probably out of memory.\");\n *init_me = (apop_name){ };\n\treturn init_me;\n}\n\n//Currently identical to apop_settings_hash (see notes there).\nstatic unsigned long apop_name_hash(char const *str){\n unsigned long int hash = 5381;\n char c;\n while ((c = *str++)) hash = hash*33 + c;\n return hash;\n}\n\n/** Adds a name to the \\ref apop_name structure. Puts it at the end of the given list.\n\n\\param n \tAn existing, allocated \\ref apop_name structure.\n\\param add_me \tA string. If \\c NULL, do nothing; return -1.\n\\param type \t'r': add a row name
\n'c': add a matrix column name
\n't': add a text column name
\n'h': add a title (i.e., a header).
\n'v': add (or overwrite) the vector name
\n\\return \tReturns the number of rows/cols/depvars after you have added the new one. But if \\c add_me is \\c NULL, return -1.\n*/\nint apop_name_add(apop_name * n, char const *add_me, char type){\n if (!add_me)\n return -1;\n\tif (type == 'h'){\n free(n->title);\n Asprintf(&n->title, \"%s\", add_me);\n return 1;\n\t} \n\tif (type == 'v'){\n\t\tn->vector\t= realloc(n->vector, strlen(add_me) + 1);\n\t\tstrcpy(n->vector, add_me);\n\t\treturn 1;\n\t} \n\tif (type == 'r'){\n\t\tn->rowct++;\n\t\tn->row\t= realloc(n->row, sizeof(char*) * n->rowct);\n\t\tn->row[n->rowct -1]\t= malloc(strlen(add_me) + 1);\n\t\tstrcpy(n->row[n->rowct -1], add_me);\n\t\tn->rowhash = realloc(n->rowhash, n->rowct * sizeof(unsigned long));\n n->rowhash[n->rowct-1] = apop_name_hash(add_me);\n\t\treturn n->rowct;\n\t} \n\tif (type == 't'){\n\t\tn->textct++;\n\t\tn->text\t= realloc(n->text, sizeof(char*) * n->textct);\n\t\tn->text[n->textct -1]\t= malloc(strlen(add_me) + 1);\n\t\tstrcpy(n->text[n->textct -1], add_me);\n\t\tn->texthash = realloc(n->texthash, n->textct * sizeof(unsigned long));\n n->texthash[n->textct-1] = apop_name_hash(add_me);\n\t\treturn n->textct;\n\t}\n\t//else assume (type == 'c')\n Apop_stopif(type != 'c', /*keep going.*/, \n 2,\"You gave me >%c<, I'm assuming you meant c; \"\n \" copying column names.\", type);\n\t\tn->colct++;\n\t\tn->col = realloc(n->col, sizeof(char*) * n->colct);\n\t\tn->col[n->colct -1]\t= malloc(strlen(add_me) + 1);\n\t\tstrcpy(n->col[n->colct -1], add_me);\n\t\tn->colhash = realloc(n->colhash, n->colct * sizeof(unsigned long));\n n->colhash[n->colct-1] = apop_name_hash(add_me);\n\t\treturn n->colct;\n}\n\n/** Prints the given list of names to stdout. Useful for debugging.\n\n\\param n The \\ref apop_name structure\n*/\nvoid apop_name_print(apop_name * n){\n if (!n) {\n printf(\"NULL\");\n return;\n }\n\tif (n->title) printf(\"title: %s\\n\", n->title);\n\tif (n->vector){\n\t\tprintf(\"vector:\");\n printf(\"\\t%s\\n\", n->vector);\n\t}\n\tif (n->colct > 0){\n\t\tprintf(\"column:\");\n\t\tfor (int i=0; i < n->colct; i++)\n\t\t\tprintf(\"\\t%s\", n->col[i]);\n\t\tprintf(\"\\n\");\n\t}\n\tif (n->textct > 0){\n\t\tprintf(\"text:\");\n\t\tfor (int i=0; i < n->textct; i++)\n\t\t\tprintf(\"\\t%s\", n->text[i]);\n\t\tprintf(\"\\n\");\n\t}\n\tif (n->rowct > 0){\n\t\tprintf(\"row:\");\n\t\tfor (int i=0; i < n->rowct; i++)\n\t\t\tprintf(\"\\t%s\", n->row[i]);\n\t\tprintf(\"\\n\");\n\t}\n}\n\t\n/** Free the memory used by an \\ref apop_name structure. */\nvoid apop_name_free(apop_name * free_me){\n if (!free_me) return; //only needed if users are doing tricky things like newdata = (apop_data){.matrix=...};\n\tfor (size_t i=0; i < free_me->colct; i++) free(free_me->col[i]);\n\tfor (size_t i=0; i < free_me->textct; i++) free(free_me->text[i]);\n\tfor (size_t i=0; i < free_me->rowct; i++) free(free_me->row[i]);\n if (free_me->vector) free(free_me->vector);\n\tfree(free_me->col); free(free_me->colhash);\n\tfree(free_me->text); free(free_me->texthash);\n\tfree(free_me->row); free(free_me->rowhash);\n\tfree(free_me);\n}\n\n/** Append one list of names to another.\n\nIf the first list is empty, then this is a copy function.\n\n\\param n1 The first set of names (no default, must not be \\c NULL)\n\\param nadd The second set of names, which will be appended after the first. (no default. If \\c NULL, a no-op.)\n\\param type1 Either 'c', 'r', 't', or 'v' stating whether you are merging the\ncolumns, rows, text, or vector. If 'v', then ignore \\c typeadd and just overwrite the\ntarget vector name with the source name. (default: 'r')\n\\param typeadd Either 'c', 'r', 't', or 'v' stating whether you are merging the columns, rows, or text. If 'v', then overwrite the target with the source vector name. (default: type1)\n*/\nAPOP_VAR_HEAD void apop_name_stack(apop_name * n1, apop_name *nadd, char type1, char typeadd){\n apop_name * apop_varad_var(nadd, NULL); \n if (!nadd) return;\n apop_name * apop_varad_var(n1, NULL);\n Apop_stopif(!n1, return, 0, \"Can't stack onto a NULL set of names (which n1 is).\");\n char apop_varad_var(type1, 'r');\n char apop_varad_var(typeadd, type1);\nAPOP_VAR_ENDHEAD\n int i;\n apop_name counts = (apop_name){.rowct=nadd->rowct, .textct = nadd->textct, .colct = nadd->colct};//Necessary when stacking onto self.;\n if (typeadd == 'v')\n apop_name_add(n1, nadd->vector, 'v');\n else if (typeadd == 'r')\n for (i=0; i< counts.rowct; i++)\n apop_name_add(n1, nadd->row[i], type1);\n else if (typeadd == 't')\n for (i=0; i< counts.textct; i++)\n apop_name_add(n1, nadd->text[i], type1);\n else if (typeadd == 'c')\n for (i=0; i< counts.colct; i++)\n apop_name_add(n1, nadd->col[i], type1);\n else Apop_notify(1, \"'%c' sent to apop_name_stack, but the only \"\n \"valid options are r t c v. Doing nothing.\", typeadd);\n}\n\n/** Copy one \\ref apop_name structure to another. That is, all data is duplicated.\n\nUsed internally by \\ref apop_data_copy, but sometimes useful by itself. For example,\nsay that we have an \\ref apop_data struct named \\c d and a \\ref gsl_matrix of the same\ndimensions named \\c m; we could give \\c m the labels from \\c d for printing:\n\\code\napop_data *wrapped = &(apop_data){.matrix=m, .names=apop_name_copy(d)};\napop_data_print(wrapped);\napop_name_free(wrapped->names); //wrapped itself is auto-allocated; do not free.\n\\endcode\n \n\\param in The input names\n\\return A \\ref apop_name struct with copies of all input names.\n*/\napop_name * apop_name_copy(apop_name *in){\n apop_name *out = apop_name_alloc();\n apop_name_stack(out, in, 'v');\n apop_name_stack(out, in, 'c');\n apop_name_stack(out, in, 'r');\n apop_name_stack(out, in, 't');\n Asprintf(&out->title, \"%s\", in->title);\n return out;\n}\n\n/** Finds the position of an element in a list of names.\n\nThe function uses POSIX's \\c strcasecmp, and so does case-insensitive search the way that function does.\n\n\\param n the \\ref apop_name object to search.\n\\param name the name you seek; see above.\n\\param type \\c 'c' (=column), \\c 'r' (=row), or \\c 't' (=text). Default is \\c 'c'.\n\\return The position of \\c findme. If \\c 'c', then this may be -1, meaning the vector name. If not found, returns -2. On error, e.g. name==NULL, returns -2.\n*/\nint apop_name_find(const apop_name *n, const char *name, const char type){\n Apop_stopif(!name, return -2, 0, \"You asked me to search for NULL.\");\n char **list;\n unsigned long *listh;\n int listct;\n if (type == 'r' || type == 'R'){\n list = n->row;\n listh = n->rowhash;\n listct = n->rowct;\n }\n else if (type == 't' || type == 'T'){\n list = n->text;\n listh = n->texthash;\n listct = n->textct;\n }\n else { // default type == 'c'\n list = n->col;\n listh = n->colhash;\n listct = n->colct;\n }\n\n if (listh) {\n unsigned long hash = apop_name_hash(name);\n for (int i = 0; i < listct; i++)\n if (hash==listh[i] && !strcasecmp(name, list[i]))\n return i;\n }\n\n //Hashes may be broken, so try again with plain string comparisons.\n for (int i = 0; i < listct; i++)\n if (!strcasecmp(name, list[i])) return i;\n\n if ((type=='c' || type == 'C') && n->vector && !strcasecmp(name, n->vector)) return -1;\n return -2;\n}\n" }, { "path": "apop_output.m4.c", "content": "/** \\file \n Some printing and output interface functions. */\n/* Copyright (c) 2006--2007, 2009 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n//The reader will find a few function headers for this file in asst.h\n#include \"apop_internal.h\"\n \n#define Output_vars output_name, output_pipe, output_type, output_append\n\n#define Output_declares char const * output_name, FILE * output_pipe, char output_type, char output_append\n\n/** If you're reading this, it is probably because you were referred by another function\n that uses this internally. You should never call this function directly, but do read\n this documentation.\n\n There are four settings that affect how output happens, which can be set when you call the\n function that sent you to this documentation, e.g:\n\n \\code\n apop_data_print(your_data, .output_type ='f', .output_append = 'w');\n \\endcode\n\n \\param output_name The name of the output file, if any. For a database, the table to write.\n \\param output_pipe If you have already opened a file and have a \\c FILE* on hand, use\n this instead of giving the file name.\n \\param output_type \\c 'p' = pipe, \\c 'f'= file, \\c 'd' = database\n \\param output_append \\c 'a' = append (default), \\c 'w' = write over.\n\nAt the end, \\c output_name, \\c output_pipe, and \\c output_type are all set.\nNotably, the local \\c output_pipe will have the correct location for the calling function to \\c fprintf to.\n\n\\li See \\ref legi for more discussion.\n\n\\li The default is output to stdout. For example,\n\\code\napop_data_print(your_data);\n//is equivalent to\napop_data_print(your_data, .output_type='p', .output_pipe=stdout);\n\\endcode\n\n\\li Tip: if writing to the database, you can get a major speed boost by wrapping the call in a begin/commit wrapper:\n\n\\code\napop_query(\"begin;\");\napop_data_print(your_data, .output_name=\"dbtab\", .output_type='d');\napop_query(\"commit;\");\n\\endcode\n\\ingroup all_public\n*/\nint apop_prep_output(char const *output_name, FILE ** output_pipe, char *output_type, char *output_append){\n *output_append = *output_append ? *output_append : 'w';\n\n if (!output_name && !*output_pipe && !*output_type) *output_type = 's'; \n else if (output_name && !*output_pipe && !*output_type) *output_type = 'f'; \n else if (!output_name && *output_pipe && !*output_type) *output_type = 'p'; \n\n if (*output_type =='p') *output_pipe = *output_pipe ? *output_pipe: stdout; \n else if (*output_type =='d') *output_pipe = stdout; //won't be used.\n else *output_pipe = output_name\n ? fopen(output_name, *output_append == 'a' ? \"a\" : \"w\")\n : stdout;\n Apop_stopif(!output_pipe && output_name, return -1, 0, \"Trouble opening file %s.\", output_name);\n return 0;\n}\n\n#define Dispatch_output \\\n char const *apop_varad_var(output_name, NULL); \\\n FILE * apop_varad_var(output_pipe, NULL); \\\n char apop_varad_var(output_type, 0); \\\n char apop_varad_var(output_append, 0); \\\n Apop_stopif(apop_prep_output(output_name, &output_pipe, &output_type, &output_append), \\\n return, 0, \"Trouble preparing to write output.\");\n\n\n/////The printing functions.\n\nstatic void white_pad(int ct){\n for(size_t i=0; i < ct; i ++)\n printf(\" \");\n}\n\n/* This function prettyprints the \\c apop_data set to a screen.\n\nIt is currently not in the documentation. It'd be nice to merge this w/apop_data_print.\n\nThis takes a lot of machinery. I write every last element to a text array, then measure column widths, then print to screen with padding to guarantee that everything lines up. There's no way to have the first element of a column line up with the last unless you interrogate the width of every element in the column, so printing columns really can't be a one-pass process.\n\nSo, I produce an \\ref apop_data set with no numeric elements and a text element to be\nfilled with the input data set, and then print that. That means that I'll be using\n(more than) twice the memory to print this. If this is a problem, you can use \\ref\napop_data_print to dump your data to a text file, and view the text file, or print\nsubsets.\n\nFor more machine-readable printing, see \\ref apop_data_print.\n*/\nvoid apop_data_show(const apop_data *in){\n if (!in) {printf(\"NULL\\n\"); return;}\n Get_vmsizes(in) //vsize, msize1, msize2, tsize\n//Take inventory and get sizes\n size_t hasrownames = (in->names && in->names->rowct) ? 1 : 0;\n size_t hascolnames = in->names && \n (in->names->vector || in->names->colct || in->names->textct);\n size_t hasweights = (in->weights != NULL);\n\n size_t outsize_r = GSL_MAX(in->matrix ? in->matrix->size1 : 0, in->vector ? in->vector->size: 0);\n outsize_r = GSL_MAX(outsize_r, in->textsize[0]);\n outsize_r = GSL_MAX(outsize_r, wsize);\n if (in->names) outsize_r = GSL_MAX(outsize_r, in->names->rowct);\n outsize_r += hascolnames;\n\n size_t outsize_c = msize2;\n outsize_c += in->textsize[1];\n outsize_c += (vsize>0);\n outsize_c += (wsize>0);\n outsize_c += hasrownames + hasweights;\n\n//Write to the printout data set.\n apop_data *printout = apop_text_alloc(NULL , outsize_r, outsize_c);\n if (hasrownames)\n for (size_t i=0; i < in->names->rowct; i ++)\n apop_text_set(printout, i + hascolnames, 0, \"%s\", in->names->row[i]);\n for (size_t i=0; i < vsize; i ++) //vsize may be zero.\n apop_text_set(printout, i + hascolnames, hasrownames, \"%g\", gsl_vector_get(in->vector, i));\n for (size_t i=0; i < msize1; i ++) //msize1 may be zero.\n for (size_t j=0; j < msize2; j ++)\n apop_text_set(printout, i + hascolnames, hasrownames + (vsize >0)+ j, \"%g\", gsl_matrix_get(in->matrix, i, j));\n if (in->textsize[0])\n for (size_t i=0; i < in->textsize[0]; i ++)\n for (size_t j=0; j < in->textsize[1]; j ++)\n apop_text_set(printout, i + hascolnames, hasrownames + (vsize>0)+ msize2 + j, \"%s\", in->text[i][j]);\n if (hasweights)\n for (size_t i=0; i < in->weights->size; i ++)\n apop_text_set(printout, i + hascolnames, outsize_c-1, \"%g\", gsl_vector_get(in->weights, i));\n\n//column names\n if (hascolnames){\n if (vsize && in->names->vector)\n apop_text_set(printout, 0 , hasrownames, \"%s\", in->names->vector);\n if (msize2 && in->names)\n for (size_t i=0; i < in->names->colct; i ++)\n apop_text_set(printout, 0 , hasrownames + (vsize>0) + i, \"%s\", in->names->col[i]);\n if (in->textsize[1] && in->names)\n for (size_t i=0; i < in->names->textct; i ++)\n apop_text_set(printout, 0 , hasrownames + (vsize>0) + msize2 + i, \"%s\", in->names->text[i]);\n if (hasweights)\n apop_text_set(printout, 0 , outsize_c-1, \"Weights\");\n }\n\n//get column sizes\n int colsizes[outsize_c];\n for (size_t i=0; i < outsize_c; i ++){\n colsizes[i] = strlen(printout->text[0][i]);\n for (size_t j=1; j < outsize_r; j ++)\n colsizes[i] = GSL_MAX(colsizes[i], strlen(printout->text[j][i]));\n }\n\n//Finally, print\n if (in->names && in->names->title && strlen(in->names->title))\n printf(\"\\t%s\\n\\n\", in->names->title);\n for (size_t j=0; j < outsize_r; j ++){\n for (size_t i=0; i < outsize_c; i ++){\n white_pad(colsizes[i] - strlen(printout->text[j][i]) + 1);//one spare space.\n printf(\"%s\", printout->text[j][i]);\n if (i > 0 && i< outsize_c-1) \n printf(\" %s \", apop_opts.output_delimiter);\n }\n printf(\"\\n\");\n }\n\n if (in->more) {\n printf(\"\\n\");\n apop_data_show(in->more);\n }\n apop_data_free(printout);\n}\n\nvoid p_fn(FILE * f, double data){\n if (data == (int) data) fprintf(f, \"% 5i\", (int) data); \n else fprintf(f, \"% 5f\", data);\n}\n\nstatic void print_core_v(const gsl_vector *data, char *separator, Output_declares){\n FILE *f = output_pipe;\n if (!data) fprintf(f, \"NULL\\n\");\n else {\n\t for (size_t i=0; isize; i++){\n\t\t p_fn(f, gsl_vector_get(data, i));\n\t\t if (i< data->size -1) fprintf(f, \"%s\", separator);\n\t }\n\t fprintf(f,\"\\n\");\n }\n\tif (output_name) fclose(f);\n}\n\n/** Print a vector to the screen, a file, a pipe, or the database.\n\n \\li See \\ref apop_prep_output for more on how printing settings are set.\n \\li For example, the default for \\ref apop_opts_type \"apop_opts.output_delimiter\"\nis a tab, which puts the vector on one line, but apop_opts.output_type=\"\\n\"\nwould print the vector vertically.\n \\li See also \\ref Legi for more details and examples.\n \\li This function uses the \\ref designated syntax for inputs.\n\\ingroup all_public\n*/\nAPOP_VAR_HEAD void apop_vector_print(gsl_vector *data, Output_declares){\n gsl_vector *apop_varad_var(data, NULL);\n Dispatch_output\nAPOP_VAR_ENDHEAD\n\tprint_core_v(data, apop_opts.output_delimiter, Output_vars);\n }\n\n/* currently removed from the documentation.\n Dump a gsl_vector to the screen. \n You may want to set \\ref apop_opts_type \"apop_opts.output_delimiter\".\n\n\\li See \\ref apop_prep_output for more on how printing settings are set.\n\\li See also \\ref Legi for more details and examples.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nvoid apop_vector_show(const gsl_vector *data){\n\tprint_core_v(data, apop_opts.output_delimiter, NULL, stdout, 's', 0); \n}\n\nstatic int get_max_strlen(char **names, size_t len){\n int max = 0;\n for (int i=0; i< len; i++)\n max = GSL_MAX(max, strlen(names[i]));\n return max;\n}\n\n//On screen, display a pipe, else use the usual output delimiter.\nstatic void a_pipe(FILE *f, char displaytype){\n if (displaytype == 's') fprintf(f, \" | \");\n else fprintf(f, \"%s\", apop_opts.output_delimiter);\n}\n\nstatic void apop_data_print_core(const apop_data *data, FILE *f, char displaytype){\n if (!data){\n fprintf(f, \"NULL\\n\");\n return;\n }\n int i, j, L = 0, \n start = (data->vector)? -1 : 0,\n end = (data->matrix)? data->matrix->size2 : 0,\n rowend = (data->matrix)? data->matrix->size1 : (data->vector) ? data->vector->size : data->text ? data->textsize[0] : -1;\n if (data->names && data->names->title && strlen(data->names->title))\n fprintf(f, \"\\t%s\\n\\n\", data->names->title);\n if (data->names && data->names->rowct)\n L = get_max_strlen(data->names->row, data->names->rowct);\n if (data->names && data->names->rowct && (data->names->vector || data->names->colct || data->names->textct)){\n if ((apop_opts.db_name_column || *apop_opts.db_name_column=='\\0') || \n !strcmp(apop_opts.db_name_column, \"row_names\"))\n fprintf(f, \"%*s \", L+2, \" \");\n else { fprintf(f, \"%s\", apop_opts.db_name_column); a_pipe(f, displaytype); }\n }\n if (data->vector && data->names && data->names->vector){\n fprintf(f, \"%s\", data->names->vector);\n }\n if (data->matrix){\n if (data->vector && data->names && data->names->colct){\n fprintf(f, \"%c \", data->names->vector ? ' ' : '\\t' );\n a_pipe(f, displaytype);\n }\n if (data->names) \n for(i=0; i< data->names->colct; i++){\n if (i < data->names->colct -1)\n fprintf(f, \"%s%s\", data->names->col[i], apop_opts.output_delimiter);\n else\n fprintf(f, \"%s\", data->names->col[i]);\n }\n }\n if (data->textsize[1] && data->names && data->names->textct){\n if ((data->vector && data->names && data->names->vector) || (data->matrix && data->names->colct))\n a_pipe(f, displaytype);\n if (data->names)\n for(i=0; i< data->names->textct; i++){\n if (i < data->names->textct -1)\n fprintf(f, \"%s%s\", data->names->text[i], apop_opts.output_delimiter);\n else\n fprintf(f, \"%s\", data->names->text[i]);\n }\n }\n if(data->names && (data->names->vector || data->names->colct || data->names->textct))\n fprintf(f, \"\\n\");\n for(j=0; j< rowend; j++){\n if (data->names && data->names->rowct > j)\n fprintf(f, \"%*s%s\", L+2, data->names->row[j], apop_opts.output_delimiter);\n for(i=start; i< end; i++){\n if ((i < 0 && j < data->vector->size) || (i>= 0 && j < data->matrix->size1 && i < data->matrix->size2))\n p_fn(f, apop_data_get(data, j, i));\n else\n fprintf(f, \" \");\n if (i==-1 && data->matrix) \n a_pipe(f, displaytype);\n if (i < end-1)\n fprintf(f, \"%s\", apop_opts.output_delimiter);\n }\n if (data->text){\n if (data->vector || data->matrix)\n a_pipe(f, displaytype);\n if (j < data->textsize[0])\n for(i=0; i< data->textsize[1]; i++){\n fprintf(f, \"%s\", data->text[j][i]);\n if (i < data->textsize[1]-1) fprintf(f, \"%s\", apop_opts.output_delimiter);\n }\n }\n if (data->weights && j < data->weights->size){\n a_pipe(f, displaytype);\n p_fn(f, data->weights->data[j]);\n }\n fprintf(f, \"\\n\");\n }\n}\n\n/** Print an \\ref apop_data set to a file, the database, or the screen,\n as determined by the \\c .output_type.\n\n\\li See \\ref apop_prep_output for more on how printing settings are set.\n\\li See \\ref Legi for more details and examples.\n\\li See \\ref sqlsec for notes on writing an \\ref apop_data set to the database.\n\\li This function uses the \\ref designated syntax for inputs.\n\\ingroup all_public\n*/\nAPOP_VAR_HEAD void apop_data_print(const apop_data *data, Output_declares){\n const apop_data * apop_varad_var(data, NULL);\n Dispatch_output\nAPOP_VAR_ENDHEAD \n if (output_type == 'd'){\n if (output_append == 'w') apop_table_exists(output_name, 'd');\n apop_data_to_db(data, output_name, output_append);\n return;\n }\n apop_data_print_core(data, output_pipe, output_type);\n if (data && data->more) {\n output_append='a';\n apop_data_print(data->more, Output_vars);\n }\n if (output_name)\n fclose(output_pipe);\n}\n\n/** Print a \\c gsl_matrix to the screen, a file, a pipe, or a database table.\n\n\\li See \\ref apop_prep_output for more on how printing settings are set.\n\\li See also \\ref Legi for more details and examples.\n\\li This function uses the \\ref designated syntax for inputs.\n\\ingroup all_public\n*/\nAPOP_VAR_HEAD void apop_matrix_print(const gsl_matrix *data, Output_declares){\n const gsl_matrix *apop_varad_var(data, NULL);\n Dispatch_output\nAPOP_VAR_ENDHEAD\n if (output_type == 'd'){\n Apop_assert_c(data, , 1, \"You sent me a NULL matrix. No database table will be created.\");\n } else if (!data){\n fprintf(output_pipe, \"NULL\\n\");\n return;\n }\n apop_data_print(&(apop_data){.matrix=(gsl_matrix*)data}, Output_vars); //cheating on the const qualifier\n}\n\n//leaving this undocumented for now.\nvoid apop_matrix_show(const gsl_matrix *data){\n apop_data_print_core(&(apop_data){.matrix=(gsl_matrix*)data}, stdout, 's');\n}\n" }, { "path": "apop_rake.m4.c", "content": "//#define __USE_POSIX //for strtok_r\n#include \"apop_internal.h\"\n#include \n#include \nvoid xprintf(char **q, char *format, ...); //in apop_conversions.c\n\n/* This is the internal documentation for apop_rake(). I assume you've read the usage\n documentation already (if you haven't, it's below, just above apop_rake).\n \n I started with:\nAlgorithm AS 51 Appl. Statist. (1972), vol. 21, p. 218\noriginal (C) Royal Statistical Society 1972\n\nBut at this point, I'm not sure if any of the original code remains at all, because the\nsparse method and the full-matrix method differ so much. \n\nThere are two phases to the process: the SQL part and the in-memory table part. The file\nhere begins with the indexing of the in-memory table. The index on the in-memory\ntable is somewhat its own structure, with get/set functions and so forth, so it has\nits own section, listed first in this file. After that, we get to the raking function\n(c_loglin) and its supporting functions, and then the apop_rake function itself. The\nSQL work itself is inside apop_rake, and c_loglin is called at the end to do the\nraking. */\n\n\n/* This section indexes a PMF-type apop_data struct. The index is held in a 2-d grid\nof nodes. The index's first index is the dimension, and will hold one column for each\ncolumn in the original data set. The index's second index goes over all of the values\nin the given column.\n\nPS: this was intended to one day become a general-purpose index for PMFs; making that\nhappen remains on the to-do list.\n\nmnode[i] = a dimension row\nmnode[i][j] = a value in a given dimension\nmnode[i][j].margin_ptrs = a list of all of the rows in the data set with the given value.\nmnode[i][j].margin_ptrs[k] = the kth item for the value.\n */\n\n/** \\cond doxy_ignore */\ntypedef struct {\n double val;\n bool *margin_ptrs, *fit_ptrs;\n} mnode_t;\n\ntypedef void(*index_apply_f)(mnode_t * const * const, int, void*);\n/** \\endcond */\n\nstatic int find_val(double findme, mnode_t *nodecol){\n for (int i=0; nodecol[i].val <= findme || gsl_isnan(findme); i++)\n if (nodecol[i].val == findme || (gsl_isnan(findme) && gsl_isnan(nodecol[i].val)))\n return i;\n return -1;\n}\n\n//returns -1 if the given row/col doesn't exist.\nint index_add_node(mnode_t **mnodes, size_t dim, size_t row, double val, bool is_margin){\n int index = find_val(val, mnodes[dim]);\n if (index == -1) return -1;\n if (is_margin) mnodes[dim][index].margin_ptrs[row] = true;\n else mnodes[dim][index].fit_ptrs[row] = true;\n return 0;\n}\n\nmnode_t **index_generate(apop_data const *in, apop_data const *in2){\n size_t margin_ct = in->matrix->size2;\n size_t margin_rows = in->matrix->size1;\n size_t fit_rows = in2->matrix->size1;\n mnode_t **mnodes = malloc(sizeof(mnode_t*)*(margin_ct+1));\n //allocate every node\n for(size_t i=0; i < margin_ct; i ++){\n gsl_vector *vals = apop_vector_unique_elements(Apop_cv(in, i));\n mnodes[i] = malloc(sizeof(mnode_t)*(vals->size+1));\n for(size_t j=0; j < vals->size; j ++)\n mnodes[i][j] = (mnode_t) {.val = gsl_vector_get(vals, j),\n .margin_ptrs = calloc(margin_rows, sizeof(bool)),\n .fit_ptrs = calloc(fit_rows, sizeof(bool))\n };\n mnodes[i][vals->size] = (mnode_t) {.val = GSL_POSINF}; //end-of-array sentinel\n gsl_vector_free(vals);\n }\n mnodes[margin_ct] = NULL; //end-of-array sentinel\n //put data from the matrix into the right pigeonhole\n for(size_t i=0; i < margin_ct; i++){\n for(size_t j=0; j < in->matrix->size1; j++)\n Apop_stopif(index_add_node(mnodes, i, j, apop_data_get(in, j, i), true) == -1, return mnodes,\n 0, \"I can't find a value, %g, that should've already been inserted.\", apop_data_get(in, j, i));\n for(size_t j=0; j < in2->matrix->size1; j++) //these values may not be present, in which case ignore them.\n index_add_node(mnodes, i, j, apop_data_get(in2, j, i), false);\n }\n return mnodes;\n}\n\nvoid index_free(mnode_t **in){\n\tfor (int i=0; in[i]; i++){\n\t\tfor (int j=0; !isinf(in[i][j].val); j++){\n\t\t\tfree(in[i][j].margin_ptrs);\n\t\t\tfree(in[i][j].fit_ptrs);\n }\n free(in[i]);\n\t}\n free(in);\n}\n\n/* The next two functions are a recursive iteration over all combinations of values\nfor a given index (which may be the whole thing or a margin). index_foreach does the\ninitialization of some state variables; value_loop does the odometer-like recursion. At\neach step, value_loop will either increment the current dimension's index, or if the\ncurrent index is at its limit, will loop back to zero on this index and then set the\nnext dimension as active. */\nstatic void value_loop(mnode_t *icon[], int *indices, mnode_t **values, \n int this_dim, index_apply_f f, int *ctr, void *args){\n while (!isinf(icon[this_dim][indices[this_dim]].val)){\n values[this_dim] = &icon[this_dim][indices[this_dim]];\n\t\tif (icon[this_dim+1]){\n\t\t\tindices[this_dim+1]=0;\n\t\t\tvalue_loop(icon, indices, values, this_dim+1, f, ctr, args);\n\t\t} else{\n\t\t\tf(values, *ctr, args);\n\t\t\t(*ctr)++;\n\t\t}\n indices[this_dim]++;\n\t}\n}\n\n/* Inputs: the index to be iterated over, the function to apply to each combination,\nand a void* with other sundry arguments to the function. I'll apply the function \nto an mnode_t* with a single mnode_t for each dimension. */\nvoid index_foreach(mnode_t *index[], index_apply_f f, void *args){\n int j, ctr=0;\n for (j=0; index[j]; ) j++;\n mnode_t *values[j+1];\n for (j=0; index[j]; j++) \n values[j] = &index[j][0];\n values[j] = malloc(sizeof(mnode_t));\n values[j]->val = GSL_POSINF;\n int indices[j];\n memset(indices, 0, sizeof(int)*j);\n value_loop(index, indices, values, 0, f, &ctr, args);\n free(values[j]);\n}\n\n/* Check whether an observation falls into all of the given margins.\n\nThis is for a contrast. We've already calculated the in/out vector for each value on\neach margin, and now need to && each dimension into one vector.\n\n\\param index Is actually a partial index: for each dimension, there should be only one value. Useful for the center of an index_foreach loop.\n\\param d Should already be allocated to the right size, may be filled with garbage.\n\\return d will be zero or one to indicate which rows of the indexed data set meet all criteria. */\nvoid index_get_element_list(mnode_t *const * index, bool *d, size_t len, bool is_margin){\n memcpy(d, is_margin ? index[0]->margin_ptrs: index[0]->fit_ptrs, len * sizeof(bool));\n for(size_t i=1; !isinf(index[i]->val); i++){\n bool *ptr_list = is_margin ? index[i]->margin_ptrs: index[i]->fit_ptrs;\n for(size_t j=0; j < len; j++)\n d[j] &= ptr_list[j];\n }\n}\n\n////End index.c\n\n/** \\cond doxy_ignore */\ntypedef struct {\n const apop_data *indata; \n apop_data *fit; \n size_t **elmtlist;\n size_t *elmtlist_sizes;\n\tgsl_vector *indata_values;\n mnode_t **index;\n size_t ct, al;\n double *maxdev;\n} rake_t;\n/** \\endcond */\n\nstatic void rakeinfo_grow(rake_t *r){\n r->al = (r->al+1)*2;\n r->elmtlist = realloc(r->elmtlist , sizeof(size_t*) * r->al);\n r->elmtlist_sizes = realloc(r->elmtlist_sizes, sizeof(size_t) * r->al);\n r->indata_values = apop_vector_realloc(r->indata_values, r->al);\n}\n\nstatic void rakeinfo_free(rake_t r){\n #define free_and_clear(in) free(in), (in) = NULL\n for (int i=0; i < r.ct; i++)\n free_and_clear(r.elmtlist[i]);\n free_and_clear(r.index); //these are just pointers to the main index.\n free_and_clear(r.elmtlist_sizes);\n free_and_clear(r.elmtlist);\n gsl_vector_free(r.indata_values);\n r.indata_values= NULL;\n}\n\nstatic void scaling(size_t const *elmts, size_t const n, gsl_vector *weights, double const in_sum, double *maxdev){\n double fit_sum = 0, out_sum=0;\n for(size_t i=0; i < n; i ++)\n fit_sum += weights->data[elmts[i]];\n if (!fit_sum) return; //can happen if init table is very different from margins.\n for(size_t i=0; i < n; i ++){\n out_sum +=\n weights->data[elmts[i]] *= in_sum/fit_sum;\n }\n *maxdev = GSL_MAX(fabs(fit_sum - out_sum), *maxdev);\n}\n\n/* Given one set of values from one margin, do the actual scaling.\n On the first pass, this function takes notes on each margin's element list and total \n in the original data. Later passes just read the notes and call the scaling() function above.\n*/\nstatic void one_set_of_values(mnode_t *const * const margincons, int ctr, void *in){\n rake_t *r = in;\n size_t marginsize = r->indata->matrix->size1;\n size_t fitsize = r->fit->matrix->size1;\n bool melmts[marginsize];\n bool fitelmts[fitsize];\n\tbool first_pass = false;\n double in_sum;\n\tif (ctr < r->ct)\n\t\tin_sum = gsl_vector_get(r->indata_values, ctr);\n else {\n r->ct++;\n if (ctr >= r->al || r->al==0) rakeinfo_grow(r);\n \t\tindex_get_element_list(margincons, melmts, marginsize, true);\n \t\tindex_get_element_list(margincons, fitelmts, fitsize, false);\n in_sum = 0;\n int n=0, al=0;\n r->elmtlist[ctr] = NULL;\n //use margin index to get total for this margin.\n for(int m=0; m < marginsize; m++)\n if (melmts[m]) in_sum += r->indata->weights->data[m];\n //use fit index to get elements involved in this margin\n for(int m=0; m < fitsize; m++)\n if (fitelmts[m]){\n if (n >= al) {\n al = (al+1)*2;\n r->elmtlist[ctr] = realloc(r->elmtlist[ctr], al*sizeof(size_t));\n }\n r->elmtlist[ctr][n++] = m;\n }\n r->elmtlist_sizes[ctr] = n;\n r->indata_values->data[ctr] = in_sum;\n\t\tfirst_pass = true;\n\t}\n if (!r->elmtlist_sizes[ctr]) return;\n if (!first_pass && !in_sum) return;\n scaling(r->elmtlist[ctr], r->elmtlist_sizes[ctr], r->fit->weights, in_sum, r->maxdev);\n}\n\n/* For each configuration margin, for each combination for that margin, \n call the above one_set_of_values() function. */\nstatic void main_loop(int config_ct, rake_t *rakeinfo, int k){\n for (size_t i=0; i < config_ct; i ++)\n\t\tif (k==1) index_foreach(rakeinfo[i].index, one_set_of_values, rakeinfo+i);\n\t\telse\n\t\t\tfor(int m=0; m < rakeinfo[i].ct; m++)\n\t\t\t\tone_set_of_values(NULL, m, rakeinfo+i);\n}\n\n/* Following the FORTRAN, 1 contrast ==> icon. Here, icon will be a\nsubset of the main index including only the columns pertaining to a given margin. */\nvoid generate_margin_index(mnode_t **icon, const apop_data *margin, mnode_t **mainindex, size_t col){\n gsl_vector *iconv = Apop_cv(margin, col);\n int ct = 0;\n for (int j=0; mainindex[j]; j++)\n if (gsl_vector_get(iconv, j))\n icon[ct++] = mainindex[j];\n icon[ct] = NULL;\n}\n\nvoid cleanup(mnode_t **index, rake_t rakeinfos[], int contrast_ct){\n\tfor(size_t i=0; i < contrast_ct; i++)\n\t\trakeinfo_free(rakeinfos[i]);\n\tindex_free(index);\n}\n\n/*\n\\param config \tAn nvar x ncon matrix; see below. [as in the original, but not squashed into 1-D.]\n\\param indata \tthe actual table. I use a PMF format.\n\\param fit \t\tthe starting table. Same size as table.\n\\param maxdev \tmaximum deviation; stop when this is met.\n\\param maxit \tmaximum iterations; stop when this is met.\n \nRe: the contrast array: Each _column_ is a contrast. Put a one in each col\n involved in the contrast. E.g., for (1,2), (2,3):\n\n 1 0
\n 1 1
\n 0 1\n */\nstatic void c_loglin(const apop_data *config, const apop_data *indata, \n apop_data *fit, double tolerance, int maxit) {\n mnode_t ** index = index_generate(indata, fit);\n\n /* Make a preliminary adjustment to obtain the fit to an empty configuration list */\n //fit->weights is either all 1 (for synthetic data) or the initial counts from the db.\n double x = apop_sum(indata->weights);\n double y = apop_sum(fit->weights);\n gsl_vector_scale(fit->weights, x/y);\n\n\tint contrast_ct =config && config->matrix ? config->matrix->size2 : 0;\n rake_t rakeinfos[contrast_ct];\n double maxdev=0;\n for(size_t i=0; i < contrast_ct; i ++){\n gsl_vector *iconv = Apop_cv(config, i);\n rakeinfos[i] = (rake_t) {\n .indata = indata, \n .fit = fit, \n .maxdev = &maxdev, //one value shared across dimensions\n .index = malloc(sizeof(mnode_t) *(apop_sum(iconv)+1)),\n //others are NULL, to be filled in as we go.\n };\n generate_margin_index(rakeinfos[i].index, config, index, i);\n }\n int k;\n gsl_vector *previous = apop_vector_copy(fit->weights);\n for (k = 1; k <= maxit; ++k) {\n maxdev = 0;\n main_loop(contrast_ct, rakeinfos, k);\n Apop_notify(3, \"Data set after round %i of raking.\\n\", k);\n if (apop_opts.verbose >=3) apop_data_print(fit, .output_pipe=apop_opts.log_file);\n if (maxdev < tolerance) break;// Normal termination \n gsl_vector_memcpy(previous, fit->weights);\n }\n cleanup(index, rakeinfos,contrast_ct);\n gsl_vector_free(previous);\n Apop_stopif(k == maxit, fit->error='c', 0, \"Maximum number of iterations reached.\");\n}\n\nchar *pipe_parse = \"[ \\n\\t]*([^| \\n\\t]+)[ \\n\\t]*([|]|$)\";\n\napop_data **generate_list_of_contrasts(char *const *contras_in, int contrast_ct){\n apop_data** out = malloc(sizeof(apop_data*)* contrast_ct);\n\tfor (int i=0; i< contrast_ct; i++) {\n apop_regex(contras_in[i], pipe_parse, out+i);\n apop_text_alloc(out[i], *out[i]->textsize, 1);\n }\n\treturn out;\n}\n\napop_data *get_var_list(char const *margin_table, char const *count_col, char const *init_count_col,\n char * const *varlist, int *var_ct){\n apop_data *all_vars_d=NULL;\n if (!varlist){\n Apop_stopif(apop_opts.db_engine=='m', apop_return_data_error(y),\n 0, \"I need a list of the full set of variable \"\n \"names sent as .varlist= (char *[]){\\\"var1\\\", \\\"var2\\\",...}\");\n //use SQLite's table_info, then shift the second col to the first.\n all_vars_d = apop_query_to_text(\"PRAGMA table_info(%s)\", margin_table);\n int ctr=0;\n for (int i=0; i< all_vars_d->textsize[0]; i++)\n if (all_vars_d->text[i][1] && (count_col ? strcmp(all_vars_d->text[i][1], count_col) : 1)\n && (init_count_col ? strcmp(all_vars_d->text[i][1], init_count_col): 1))\n apop_text_set(all_vars_d, 0, ctr++, all_vars_d->text[i][1]);\n apop_text_alloc(all_vars_d, 1, ctr);\n *var_ct = ctr;\n } else {\n all_vars_d = apop_text_alloc(NULL, 1, *var_ct); \n for (int i=0; i<*var_ct; i++) apop_text_set(all_vars_d, 0, i, varlist[i]);\n }\n Apop_stopif(!all_vars_d, apop_return_data_error(y), 0, \"Trouble getting/parsing the list of variables.\");\n return all_vars_d;\n}\n\nstatic int get_var_index(char *const *all_vars, int len, char *findme){\n\tfor (int i=0; i< len; i++)\n\t\tif (all_vars[i] && !strcmp(all_vars[i], findme))\n\t\t\treturn i;\n\tApop_notify(0, \"I couldn't find %s in the full list of variables. Returning -1.\", findme);\n return -1;\n}\n\nvoid nan_to_zero(double *in){ if (gsl_isnan(*in)) *in=0;}\n\ndouble nudge_zeros(apop_data *in, void *nudge){\n if (!in->weights->data[0])\n in->weights->data[0] = *(double*)nudge;\n return 0;\n}\n\nint find_in_allvars(char const *in, apop_data const *allvars){\n for (int i=0; i< allvars->textsize[1]; i++)\n if (!strcmp(allvars->text[0][i], in)) return i;\n Apop_stopif(1, return -1, 0, \"Variable in your contrast list [%s] not in \"\n \"your list of all variables.\", in);\n}\n\n/* If you are fully synthesizing or nudging zero cells, then calculate the set of \n cells that could be nonzero (given the margin information). \n * If using only an init_table, then that's your list of nonzero cells right there.\n * If you have an init_table but want to nudge the zero cells up a bit, then you need\n this, and have to merge with the init_table\n * If you have no init_table, then this list is all the cells.\n \n */\nstatic int setup_nonzero_contrast(char const *margin_table, \n apop_data const * allvars, int run_number,\n char const *list_of_fields, apop_data *const* contras, int contrast_ct,\n double nudge, char const * structural_zeros, bool have_init_table){\n char *q;\n bool used[allvars->textsize[1]];\n memset(used, 0, sizeof(bool)*allvars->textsize[1]);\n\tAsprintf(&q, \"create table apop_zerocontrasts_%i as select %s, %g from\\n\", \n run_number, apop_text_paste(allvars, .between=\",\"), nudge);\n for (int i=0; i < contrast_ct; i++){\n xprintf(&q, \"%s%s (select distinct %s from %s) \\n\", \n q, i>0 ? \"natural join\" : \"\", \n apop_text_paste(contras[i], .between=\",\"), margin_table);\n for (int j=0; j<*contras[i]->textsize; j++){\n int val=find_in_allvars(*contras[i]->text[j], allvars);\n Apop_stopif(val==-1, return -1, 0, \"Error setting up contrasts\");\n used[val]=true; \n }\n } \n //make sure all variables are joined in. \n for (int i=0; i < allvars->textsize[1]; i++)\n if (!used[i]) xprintf(&q, \"%s%s (select distinct %s from %s)\\n\", \n q, (!contrast_ct && i==0) ? \"\" : \"natural join\",\n allvars->text[0][i], margin_table);\n if (structural_zeros) xprintf(&q, \"%s where not (%s)\\n\", q, structural_zeros);\n if (have_init_table){\n //Keep out margins with values for now; join them in below.\n xprintf(&q, \"%s except\\nselect %s, %g from %s\", q, list_of_fields, nudge, margin_table);\n }\n\tApop_stopif(apop_query(\"%s\", q), return 1, 0, \"query failed.\");\n free(q);\n return 0;\n}\n\n/** Fit a log-linear model via iterative proportional fitting, aka raking.\n\nRaking has many uses. The Modeling with Data blog presents a series of discussions \nof uses of raking, including some worked examples.\n\nOr see Wikipedia for an overview of Log linear models, aka\nPoisson regressions. \nOne approach toward log-linear modeling is a regression form; let there be four\ncategories, A, B, C, and D, from which we can produce a model positing, for example,\nthat cell count is a function of a form like \\f$g_1(A) + g_2(BC) + g_3(CD)\\f$. In this case, we would\nassign a separate coefficient to every possible value of A, every possible value of\n(B, C), and every value of (C, D). Raking is the technique that searches for that large\nset of parameters.\n\nThe combinations of categories that are considered to be relevant are called \\em\ncontrasts, after ANOVA terminology of the 1940s.\n\nThe other constraint on the search are structural zeros, which are values that you know\ncan never be non-zero, due to field-specific facts about the variables. For example, U.S.\nSocial Security payments are available only to those age 65 or older, so \"age <65 and\ngets_soc_security=1\" is a structural zero.\n\nBecause there is one parameter for every combination, there may be millions of parameters\nto estimate, so the search to find the most likely value requires some attention to\ntechnique. For over half a century, the consensus method for searching has been raking, which\niteratively draws each category closer to the mean in a somewhat simple manner (this was\nfirst developed circa 1940 and had to be feasible by hand), but which is guaranteed to\neventually arrive at the maximum likelihood estimate for all cells.\n\nAnother complication is that the table is invariably sparse. One can easily construct\ntables with millions of cells, but the corresponding data set may have only a few\nthousand observations.\n\nThis function uses the database to resolve the sparseness problem. It constructs a query\nrequesting all combinations of categories the could possibly be non-zero after raking,\ngiven all of the above constraints. Then, raking is done using only that subset. \nThis means that the work is done on a number of cells proportional to the number of data\npoints, not to the full cross of all categories. Set apop_opts.verbose to 2 or greater to show the query on \\c stderr.\n\n\\li One could use raking to generate `fully synthetic' data: start with observation-level data in a margin table. Begin the raking with a starting data set of all-ones. Then rake until the all-ones set transforms into something that conforms to the margins and (if any) structural zeros. You now have a data set which matches the marginal totals but does not use any other information from the observation-level data. If you do not specify an .init_table, then an all-ones default table will be used.\n\n\n\\param margin_table The name of the table in the database to use for calculating\nthe margins. The table should have one observation per row. (No default)\n\n\\param var_list The full list of variables to search. A list of strings, e.g., (char *[]){\"var1\", \"var2\", ..., \"var15\"}\n\n\\param var_ct The count of the full list of variables to search.\n\n\\param contrasts The contrasts describing your model. Like the \\c var_list input, a list of strings like (char *[]){\"var1\", \"var7\", \"var13\"}\ncontrast is a pipe-delimited list of variable names. (No default)\n\n\\param contrast_ct The number of contrasts in the list of contrasts. (No default)\n\n\\param structural_zeros a SQL clause indicating combinations that can never take a nonzero\nvalue. This will go into a \\c where clause, so anything you could put there is OK, e.g.\n\"age <65 and gets_soc_security=1 or age <15 and married=1\". \nYour margin data is not checked for structural zeros. (default: no structural zeros)\n\n\\param max_iterations Number of rounds of raking at which the algorithm halts. (default: 1000)\n\n\\param tolerance I calculate the change for each cell from round to round;\nif the largest cell change is smaller than this, I stop. (default: 1e-5)\n\n\\param count_col This column gives the count of how many observations are represented\nby each row. If \\c NULL, ech row represents one person. (default: \\c NULL)\n\n\\param init_table The default is to initially set all table elements to one and then\nrake from there. This is effectively the `fully synthetic' approach, which uses only\nthe information in the margins and derives the data set closest to the all-ones data\nset that is consistent with the margins. Care is taken to maintan sparsity in this\ncase. If you specify an \\c init_table, then I will get the initial cell counts from\nit. (default: the fully-synthetic approach, using a starting point of an all-ones grid.)\n\n\\param init_count_col The column in \\c init_table with the cell counts.\n\n\\param nudge There is a common hack of adding a small value to every zero entry, because\na zero entry will always scale to zero, while a small value could eventually scale\nto anything. Recall that this function works on sparse sets, so I first filter out\nthose cells that could possibly have a nonzero value given the observations, then I\nadd nudge to any zero cells within that subset.\n\n\\return An \\ref apop_data set where every row is a single combination of variable values\nand the \\c weights vector gives the most likely value for each cell.\n\n\\exception out->error='i' Input was somehow wrong.\n\\exception out->error='c' Raking did not converge, reached max. iteration count.\n\n\\li Set apop_opts.verbose=3 to see the intermediate tables at the end of each round of raking.\n\\li If you want all cells to have nonzero value, then you can do that via pre-processing:\n\\code\napop_query(\"update data_table set count_col = 1e-3 where count_col = 0\");\n\\endcode\n\\li This function is thread-safe. To make this happen, temp database tables are \n named using a number built with \\c omp_get_thread_num.\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data * apop_rake(char const *margin_table, char * const*var_list, int var_ct, char * const *contrasts, int contrast_ct, char const *structural_zeros, int max_iterations, double tolerance, char const *count_col, char const *init_table, char const *init_count_col, double nudge){\n char const * apop_varad_var(margin_table, NULL);\n Apop_stopif(!margin_table, apop_return_data_error(i), 0, \n \"I need the name of a table in the database that will be the data source.\");\n Apop_stopif(!apop_table_exists(margin_table), apop_return_data_error(i), \n 0, \"your margin_table, %s, doesn't exist in the database.\", margin_table);\n char *const* apop_varad_var(var_list, NULL);\n int apop_varad_var(var_ct, 0);\n char *const * apop_varad_var(contrasts, NULL); //default to all vars?\n int apop_varad_var(contrast_ct, 0);\n Apop_stopif(contrasts&&!contrast_ct, apop_return_data_error(i),\n 0, \"you gave me a list of contrasts but not the count. \"\n \"This is C--I can't count them myself. Please provide the count and re-run.\");\n char const * apop_varad_var(structural_zeros, NULL);\n char const * apop_varad_var(count_col, NULL);\n int apop_varad_var(max_iterations, 1e3);\n double apop_varad_var(tolerance, 1e-5);\n char const * apop_varad_var(init_count_col, NULL);\n char const * apop_varad_var(init_table, NULL);\n Apop_stopif(init_table && !apop_table_exists(init_table), apop_return_data_error(i),\n 0, \"your init_table, %s, doesn't exist in the database.\", init_table);\n if (init_count_col && !init_table) init_table = margin_table;\n double apop_varad_var(nudge, 0);\nAPOP_VAR_ENDHEAD\n #ifdef OpenMP\n int run_number = omp_get_thread_num();\n #else\n int run_number = 0;\n #endif\n\n\tapop_data **contras = generate_list_of_contrasts(contrasts, contrast_ct);\n apop_data *all_vars_d = get_var_list(margin_table, count_col, init_count_col, var_list, &var_ct);\n Apop_stopif(all_vars_d->error, return all_vars_d, 0, \"Trouble setting up the list of variables.\");\n int tt = all_vars_d->textsize[0]; all_vars_d->textsize[0] = 1; //mask all but the first row\n char *list_of_fields = apop_text_paste(all_vars_d, .between=\", \");\n\n if (nudge || !init_table){\n char *tab;\n Asprintf(&tab, \"apop_zerocontrasts_%i\", run_number);\n apop_table_exists(tab, 'd');\n free(tab);\n Apop_stopif(setup_nonzero_contrast(margin_table, all_vars_d, \n run_number, list_of_fields, contras, contrast_ct, (nudge ? nudge : 1), structural_zeros, init_table),\n apop_return_data_error(q),\n 0, \"Couldn't calculate the set of nonzero cells.\");\n }\n\n char *initt=NULL; //handle structural zeros via subquery\n //note that margin data may have invalid rows.\n if (init_table){\n if (structural_zeros) \n Asprintf(&initt, \"(select * from %s where not (%s))\", init_table, structural_zeros);\n else Asprintf(&initt, \"%s\", init_table);\n }\n char *init_q, *pre_init_q = NULL;\n if (init_table){\n char *countstr;\n if (init_count_col) Asprintf(&countstr, \"sum(%s) as %s\", init_count_col, init_count_col);\n else Asprintf(&countstr, \"count(%s)\", **all_vars_d->text);\n Asprintf(&init_q, \"select %s, %s from %s group by %s\", \n list_of_fields, countstr, initt, list_of_fields);\n free(countstr);\n }\n\n char *marginq, *cc; \n if (count_col) Asprintf(&cc, \"sum(%s)\", count_col);\n else cc = strdup(\"count(*)\");\n Asprintf(&marginq, \"select %s, %s from %s\\ngroup by %s\", \n list_of_fields, cc, margin_table, list_of_fields);\n free(cc); free(initt);\n\n char *format=strdup(\"w\");\n for (int i =0 ; i< var_ct; i++)\n xprintf(&format, \"m%s\", format);\n apop_data *d, *contrast_grid;\n d = apop_query_to_mixed_data(format, \"%s\", marginq);\n Apop_stopif(!d || d->error, apop_return_data_error(q),\n 0, \"This query:\\n%s\\ngenerated a blank or broken table.\", marginq);\n free(marginq);\n\n if (pre_init_q) Apop_stopif(apop_query(\"%s\", pre_init_q), apop_return_data_error(q),\n 0, \"This query:\\n%s\\ngenerated a blank or broken table.\", pre_init_q);\n\n apop_data *fit;\n if (init_table) {\n fit = (nudge) \n ? apop_query_to_mixed_data(format, \"%s\\nunion\\nselect * from apop_zerocontrasts_%i \", init_q, run_number)\n : apop_query_to_mixed_data(format, \"%s\", init_q);\n Apop_stopif(!fit, apop_return_data_error(q), 0, \"Query returned a blank table.\");\n Apop_stopif(fit->error, apop_return_data_error(q), 0, \"Query error.\");\n } else {\n fit = apop_query_to_mixed_data(format, \"select * from apop_zerocontrasts_%i \", run_number);\n gsl_vector_set_all(fit->weights, nudge ? nudge : 1);\n }\n free(format);\n apop_vector_apply(fit->weights, nan_to_zero);\n if (nudge) apop_map(fit, .fn_rp=nudge_zeros, .param=&nudge);\n\n contrast_grid = apop_data_calloc(var_ct, contrast_ct);\n\tfor (int i=0; i< contrast_ct; i++)\n\t\tfor (int j=0; j< contras[i]->textsize[0]; j++)\n\t\t\tapop_data_set(contrast_grid, get_var_index(*all_vars_d->text, all_vars_d->textsize[1], contras[i]->text[j][0]), i, 1);\n\t\n if (!init_table || nudge)\n for (int i=0; i< contrast_ct; i++) apop_data_free(contras[i]);\n\n c_loglin(contrast_grid, d, fit, tolerance, max_iterations);\n apop_data_free(d);\n \n all_vars_d->textsize[0] = tt;\n apop_data_free(all_vars_d);\n if (!init_table || nudge) apop_query(\"drop table apop_zerocontrasts_%i\", run_number);\n\tapop_data_free(contrast_grid);\n\treturn fit;\n}\n" }, { "path": "apop_regression.m4.c", "content": "/** \\file apop_regression.c\tGenerally, if it assumes something is Normally distributed, it's here.*/\n/* Copyright (c) 2006--2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include //lsearch; bsearch is in stdlib.\n\n/* For use by MLE, OLS, et al. Available for public use, but undocumented. */\nvoid apop_estimate_parameter_tests (apop_model *est){\n Nullcheck_p(est, )\n if (!est->data) return;\n apop_data *ep = apop_data_add_page(est->info, apop_data_alloc(est->parameters->vector->size, 2), \"\");\n apop_name_add(ep->names, \"p value\", 'c');\n apop_name_add(ep->names, \"confidence\", 'c');\n apop_name_stack(ep->names, est->parameters->names, 'r', 'r');\n Get_vmsizes(est->data); //msize1, vsize\n int df = msize1 ? msize1 : vsize;\n df -= est->parameters->vector->size;\n df = df < 1 ? 1 : df; //some models aren't data-oriented.\n apop_data_add_named_elmt(est->info, \"df\", df);\n\n apop_data *one_elmt = apop_data_calloc(1, 1);\n gsl_vector *param_v = apop_data_pack(est->parameters);\n for (size_t i=0; i< est->parameters->vector->size; i++){\n Apop_settings_add_group(est, apop_pm, .index=i);\n apop_model *m = apop_parameter_model(est->data, est);\n\n double zero = apop_cdf(one_elmt, m);\n apop_model_free(m);\n double conf = 2*fabs(0.5-zero); //parameter is always at 0.5 along a symmetric CDF\n apop_data_set(ep, i, .colname=\"confidence\", .val=conf);\n apop_data_set(ep, i, .colname=\"p value\", .val=1-conf);\n }\n gsl_vector_free(param_v);\n apop_data_free(one_elmt);\n}\n\n//Cut and pasted from the GNU std library documentation, modified to consider NaNs:\nstatic int compare_doubles (const void *a, const void *b) {\n const double *da = (const double *) a;\n const double *db = (const double *) b;\n if (gsl_isnan(*da)) return gsl_isnan(*db) ? 0 : 1;\n if (gsl_isnan(*db)) return -1;\n return (*da > *db) - (*da < *db);\n}\n\ntypedef const char * ccp;\nstatic int strcmpwrap(const void *a, const void *b){\n const ccp *aa = a;\n const ccp *bb = b;\n return strcmp(*aa, *bb);\n}\n\n/** Give me a vector of numbers, and I'll give you a sorted list of the unique elements.\n This is basically running select distinct datacol from data order by datacol,\n but without the aid of the database.\n\n \\param v a vector of items\n \\return a sorted vector of the distinct elements that appear in the input.\n \\li NaNs (if any) appear at the end of the sort order.\n \\see apop_text_unique_elements \n*/\ngsl_vector * apop_vector_unique_elements(const gsl_vector *v){\n size_t prior_elmt_ctr = 107;\n size_t elmt_ctr = 0;\n double *elmts = NULL;\n for (size_t i=0; i< v->size; i++){\n if (prior_elmt_ctr != elmt_ctr)\n elmts = realloc(elmts, sizeof(double)*(elmt_ctr+1));\n prior_elmt_ctr = elmt_ctr;\n double val = gsl_vector_get(v, i);\n lsearch(&val, elmts, &elmt_ctr, sizeof(double), compare_doubles);\n if (prior_elmt_ctr < elmt_ctr)\n qsort(elmts, elmt_ctr, sizeof(double), compare_doubles);\n }\n gsl_vector *out = apop_array_to_vector(elmts, elmt_ctr);\n free(elmts);\n return out;\n}\n\n/** Give me a column of text, and I'll give you a sorted list of the unique elements. \n This is basically running select distinct * from datacolumn, but without \n the aid of the database. \n\n \\param d An \\ref apop_data set with a text component\n \\param col The text column you want me to use.\n \\return An \\ref apop_data set with a single sorted column of text, where each unique text input appears once.\n \\see apop_vector_unique_elements\n*/\napop_data * apop_text_unique_elements(const apop_data *d, size_t col){\n char **tval;\n\n //first element for free\n size_t prior_elmt_ctr, elmt_ctr = 1;\n char **telmts = malloc(sizeof(char**)*2);\n telmts[0] = d->text[0][col];\n\n for (int i=1; i< d->textsize[0]; i++){\n prior_elmt_ctr = elmt_ctr;\n tval = &(d->text[i][col]);\n lsearch (tval, telmts, &elmt_ctr, sizeof(char*), strcmpwrap);\n if (prior_elmt_ctr < elmt_ctr){\n qsort(telmts, elmt_ctr, sizeof(char*), strcmpwrap);\n telmts = realloc(telmts, sizeof(char**)*(elmt_ctr+1));\n }\n }\n\n //pack and ship\n apop_data *out = apop_text_alloc(NULL, elmt_ctr, 1);\n for (int j=0; j< elmt_ctr; j++)\n apop_text_set(out, j, 0, telmts[j]);\n free(telmts);\n return out;\n}\n\nstatic char *apop_get_factor_basename(apop_data *d, int col, char type){\n char *name;\n char *catname = d->names == NULL ? NULL\n : type == 't' && d->names && d->names->textct > col ? d->names->text[col]\n : col == -1 && d->names && d->names->vector ? d->names->vector\n : col >=0 && d->names && d->names->colct > col ? d->names->col[col]\n : NULL;\n if (catname){\n Asprintf(&name, \"%s\", catname);\n return name;\n }\n if (type == 't'){\n Asprintf(&name, \"text column %i\", col);\n return name;\n }\n if (col == -1) Asprintf(&name, \"vector\");\n else Asprintf(&name, \"column %i\", col);\n return name;\n}\n\nstatic char *make_catname (apop_data *d, int col, char type){\n char *name, *subname = apop_get_factor_basename(d, col, type);\n Asprintf(&name, \"\", subname);\n free(subname);\n return name;\n}\n\n/** Factor names are stored in an auxiliary table with a name like \n\"\". Producing this name is annoying (and prevents us from eventually making it human-language independent), so use this function to get the list of factor names.\n\n\\param data The data set. (No default, must not be \\c NULL)\n\\param col The column in the main data set whose name I'll use to check for the factor name list. Vector==-1. (default=0)\n\\param type If you are referring to a text column, use 't'. (default='d')\n\n\\return A pointer to the page in the data set with the given factor names.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_data_get_factor_names(apop_data *data, int col, char type){\n apop_data *apop_varad_var(data, NULL)\n Apop_stopif(!data, return NULL, 1, \"You sent me a NULL data set. Returning NULL.\");\n int apop_varad_var(col, 0)\n char apop_varad_var(type, 'd')\nAPOP_VAR_ENDHEAD\n char *name = make_catname (data, col, type);\n apop_data *out = apop_data_get_page(data, name, .match='e');\n free(name);\n return out;\n}\n\napop_data * create_factor_list(apop_data *d, int col, char type){\n //first, create an ordered list of unique elements.\n //Record that list for use in this function, and in a ->more page of the data set.\n char *catname = make_catname(d, col, type);\n apop_data *factor_list;\n if (type == 't'){\n factor_list = apop_data_add_page(d, apop_text_unique_elements(d, col), catname);\n size_t elmt_ctr = factor_list->textsize[0];\n //awkward format conversion:\n factor_list->vector = gsl_vector_alloc(elmt_ctr);\n for (size_t i=0; i< factor_list->vector->size; i++)\n apop_data_set(factor_list, i, -1, i);\n } else {\n gsl_vector *delmts = apop_vector_unique_elements(Apop_cv(d, col));\n factor_list = apop_data_add_page(d, apop_data_alloc(), catname);\n factor_list->vector = delmts;\n apop_text_alloc(factor_list, delmts->size, 1);\n for (size_t i=0; i< factor_list->vector->size; i++){\n //shift to the text, for conformity with the more common text version.\n apop_text_set(factor_list, i, 0, \"%g\", gsl_vector_get(delmts, i));\n }\n }\n free(catname);\n return factor_list;\n}\n\n/* Producing dummies consists of finding the index of element i, for all i, then\n setting (i, index) to one.\n Producing factors consists of finding the index and then setting (i, datacol) to index.\n Otherwise the work is basically identical.\n Also, add a ->more page to the input data giving the translation.\n */\nstatic apop_data * dummies_and_factors_core(apop_data *d, int col, char type,\n int keep_first, int datacol, char dummyfactor,\n apop_data **factor_list){\n if (!(*factor_list=apop_data_get_factor_names(d, col, type)))\n *factor_list = create_factor_list(d, col, type);\n Get_vmsizes((*factor_list)); //maxsize\n size_t elmt_ctr = maxsize;\n\n //copy the strings to a single list-of-strings\n apop_data *telmts = *(*factor_list)->textsize ? apop_data_transpose(*factor_list, .inplace='n'):NULL;\n gsl_vector *delmts = (*factor_list)->vector;\n\n //Now go through the input vector, and for row i find the posn of the vector's\n //name in the element list created above (j), then change (i,j) in\n //the dummy matrix to one.\n int s = type == 't' \n ? d->textsize[0]\n : (col >=0 ? d->matrix->size1 : d->vector->size);\n apop_data *out = (dummyfactor == 'd')\n ? apop_data_calloc(0, s, (keep_first!='n' ? elmt_ctr : elmt_ctr-1))\n : d;\n size_t index;\n for (size_t i=0; i< s; i++){\n if (type == 'd'){\n double val = apop_data_get(d, i, col);\n size_t posn = (size_t)bsearch(&val, delmts->data, elmt_ctr, sizeof(double), compare_doubles);\n if (posn )\n index = (posn - (size_t)delmts->data)/sizeof(double);\n else {\n index = elmt_ctr++;\n (*factor_list)->vector = apop_vector_realloc((*factor_list)->vector, elmt_ctr);\n gsl_vector_set((*factor_list)->vector, index, val);\n out->matrix = apop_matrix_realloc(out->matrix, out->matrix->size1, elmt_ctr);\n gsl_vector_set_zero(Apop_cv(out, index));\n }\n } else {\n size_t posn = (size_t)bsearch(&(d->text[i][col]), *telmts->text, elmt_ctr, sizeof(char**), strcmpwrap);\n if (posn)\n index = (posn - (size_t)*telmts->text)/sizeof(char**);\n else {\n index = elmt_ctr++;\n *factor_list = apop_text_alloc(*factor_list, elmt_ctr, 1);\n apop_text_set(*factor_list, index, 0, d->text[i][col]);\n (*factor_list)->vector = apop_vector_realloc((*factor_list)->vector, elmt_ctr);\n apop_data_set(*factor_list, index, -1, index);\n\n telmts = apop_text_alloc(telmts, 1, elmt_ctr);\n apop_text_set(telmts, 0, index, d->text[i][col]);\n if (dummyfactor == 'd'){\n out->matrix = apop_matrix_realloc(out->matrix, out->matrix->size1, out->matrix->size2+1);\n gsl_vector_set_zero(Apop_cv(out, out->matrix->size2-1));\n }\n }\n }\n if (dummyfactor == 'd'){\n if (keep_first!='n')\n gsl_matrix_set(out->matrix, i, index,1); \n else if (index > 0) //else don't keep first and index==0; throw it out. \n gsl_matrix_set(out->matrix, i, index-1, 1); \n } else\n apop_data_set(out, i, datacol, index); \n }\n //Add names:\n if (dummyfactor == 'd'){\n char *basename = apop_get_factor_basename(d, col, type);\n for (size_t i = (keep_first!='n') ? 0 : 1; i< elmt_ctr; i++){\n char n[1000];\n if (type =='d'){\n sprintf(n, \"%s dummy %g\", basename, gsl_vector_get(delmts,i));\n } else\n sprintf(n, \"%s\", telmts->text[0][i]);\n apop_name_add(out->names, n, 'c');\n }\n }\n apop_data_free(telmts);\n return out;\n}\n\n/** A utility to make a matrix of dummy variables. You give me a single\nvector that lists the category number for each item, and I'll produce\na matrix with a single one in each row in the column specified.\n\nAfter that, you have to decide what to do with the new matrix and the original data column. \n\n\\li You can manually join the dummy data set with your main data, e.g.:\n\\code\napop_data *dummies = apop_data_to_dummies(main_regression_vars, .col=8, .type='t');\napop_data_stack(main_regression_vars, dummies, 'c', .inplace='y');\n\\endcode\n\n\\li The .remove='y' option specifies that I should use \\ref apop_data_rm_columns \nto remove the column used to generate the dummies. Implemented only for type=='d'.\n\n\\li By specifying .append='y' or .append='e' I will run the above two lines for you. Your \\ref apop_data pointer will not change, but its \\c matrix element will be reallocated (via \\ref apop_data_stack).\n\n\\li By specifying .append='i', I will place the matrix of dummies in place,\nimmediately after the data column you had specified. You will probably use this with\n.remove='y' to replace the single column with the new set of dummy columns.\nBear in mind that if there are two or more dummy columns, adding columns will change subsequent column numbers; use \\ref apop_name_find to find columns instead of giving an explicit column number.\n\n\\li If .append='i' and you asked for a text column, I will append to the end of\nthe table, which is equivalent to append='e'.\n\n\\param d The data set with the column to be dummified (No default.)\n\\param col The column number to be transformed; -1==vector (default = 0)\n\\param type 'd'==data column, 't'==text column. (default = 't')\n\\param keep_first If \\c 'n', return a matrix where each row has a one in the (column specified minus\n one). That is, the zeroth category is dropped, the first category\n has an entry in column zero, et cetera. If you don't know why this\n is useful, then this is what you need. If you know what you're doing\n and need something special, set this to \\c 'y' and the first category won't be dropped. (default = \\c 'n')\n\\param append If \\c 'e' or \\c 'y', append the dummy grid to the end of the original data\nmatrix. If \\c 'i', insert in place, immediately after the original data column. (default = \\c 'n')\n\\param remove If \\c 'y', remove the original data or text column. (default = \\c 'n')\n\n\\return An \\ref apop_data set whose \\c matrix element is the one-zero\nmatrix of dummies. If you used .append, then this is the main matrix.\nAlso, I add a page named \"\\\" giving a reference table of names and column numbers (where your_var is the appropriate column heading).\n\\exception out->error=='a' allocation error\n\\exception out->error=='d' dimension error\n\n\\li Use \\ref apop_data_get_factor_names to get the list of category names.\n\\li NaNs (if any) appear at the end of the sort order.\n\\li See \\ref fact for further discussion.\n\\li See the documentation for \\ref apop_logit for a sample linear model using this function.\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\see \\ref apop_data_to_factors\n*/\nAPOP_VAR_HEAD apop_data * apop_data_to_dummies(apop_data *d, int col, char type, int keep_first, char append, char remove){\n apop_data *apop_varad_var(d, NULL)\n Apop_stopif(!d, return NULL, 1, \"You sent me a NULL data set for apop_data_to_dummies. Returning NULL.\");\n int apop_varad_var(col, 0)\n char apop_varad_var(type, 't')\n int apop_varad_var(keep_first, 'n')\n char apop_varad_var(append, 'n')\n char apop_varad_var(remove, 'n')\n if (remove =='y' && type == 't') Apop_notify(1, \"Remove isn't implemented for text source columns yet.\");\nAPOP_VAR_ENDHEAD\n if (type == 'd'){\n Apop_stopif((col == -1) && d->vector, apop_return_data_error(d),\n 0, \"You asked for the vector element \"\n \"(col==-1) but the data's vector element is NULL.\");\n Apop_stopif((col != -1) && (col >= d->matrix->size2), apop_return_data_error(d),\n 0, \"You asked for the matrix element %i \"\n \"but the data's matrix element has only %zu columns.\", col, d->matrix->size2);\n } else Apop_stopif(col >= d->textsize[1], apop_return_data_error(d),\n 0, \"You asked for the text element %i but \"\n \"the data's text element has only %zu elements.\", col, d->textsize[1]);\n apop_data *fdummy;\n apop_data *dummies= dummies_and_factors_core(d, col, type, keep_first, 0, 'd', &fdummy);\n //Now process the append and remove options.\n size_t orig_size = d->matrix ? d->matrix->size1 : 0;\n int rm_list[orig_size+1];\n memset (rm_list, 0, (orig_size+1)*sizeof(int)); \n if (append =='i'){\n apop_data **split = apop_data_split(d, col+1, 'c');\n //stack names, then matrices\n for (int i=0; i < d->names->colct; i++)\n free(d->names->col[i]);\n apop_name_stack(d->names, split[0]->names, 'c');\n for (int k = d->names->colct; k < (split[0]->matrix ? split[0]->matrix->size2 : 0); k++)\n apop_name_add(d->names, \"\", 'c'); //pad so the name stacking is aligned (if needed)\n apop_name_stack(d->names, dummies->names, 'c');\n apop_name_stack(d->names, split[1]->names, 'c');\n gsl_matrix_free(d->matrix);\n d->matrix = apop_matrix_stack(split[0]->matrix, dummies->matrix, 'c');\n apop_data_free(dummies);\n apop_data_free(split[0]);\n apop_matrix_stack(d->matrix, split[1]->matrix, 'c', .inplace='y');\n apop_data_free(split[1]);\n return d;\n }\n if (remove!='n' && type!='t'){\n rm_list[col]=1;\n apop_data_rm_columns(d, rm_list);\n }\n if (append =='y' || append == 'e' || append ==1 || (append=='i' && type=='t')){\n d = apop_data_stack(d, dummies, 'c', .inplace='y');\n apop_data_free(dummies);\n return d;\n }\n return dummies;\n}\n\n/** Convert a column of text or numbers\n into a column of numeric factors, which you can use for a multinomial probit/logit, for example.\n\n If you don't run this on your data first, \\ref apop_probit and \\ref apop_logit default to running \n it on the vector or (if no vector) zeroth column of the matrix of the input \\ref apop_data set, because those models need a list of the unique values of the dependent variable.\n\n\\param data The data set to be modified in place. (No default. If \\c NULL, returns \\c NULL and a warning)\n\\param intype If \\c 't', then \\c incol refers to text, if \\c 'd', refers to the vector or matrix. (default = \\c 't')\n\\param incol The column in the text that will be converted. -1 is the vector. (default = 0)\n\\param outcol The column in the data set where the numeric factors will be written (-1 means the vector). (default = 0)\n\nFor example:\n\\code\napop_data *d = apop_query_to_mixed_data(\"mmt\", \"select 0, year, color from data\");\napop_data_to_factors(d);\n\\endcode\nNotice that the query pulled a column of zeros for the sake of saving room for the factors. It reads column zero of the text, and writes it to column zero of the matrix.\n\nAnother example:\n\\code\napop_data *d = apop_query_to_data(\"mmt\", \"select type, year from data\");\napop_data_to_factors(d, .intype='d', .incol=0, .outcol=0);\n\\endcode\nHere, the \\c type column is converted to sequential integer factors and\nthose factors overwrite the original data. Since a reference table is\nadded as a second page of the \\ref apop_data set, you can recover the\noriginal values as needed.\n\n\\return A table of the factors used in the code. This is an \\c apop_data set with only one column of text.\nAlso, I add a page named \"\" giving a reference table of names and column numbers (where your_var is the appropriate column heading) use \\ref apop_data_get_factor_names to retrieve that table.\n\n\\exception out->error=='a' allocation error.\n\\exception out->error=='d' dimension error.\n\n\\li If the vector or matrix you wanted to write to is \\c NULL, I will allocate it for you.\n\\li See \\ref fact for further discussion.\n\\li See the documentation for \\ref apop_logit for a sample linear model using this function.\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\see \\ref apop_data_to_dummies\n*/\nAPOP_VAR_HEAD apop_data *apop_data_to_factors(apop_data *data, char intype, int incol, int outcol){\n apop_data *apop_varad_var(data, NULL)\n Apop_stopif(!data, return NULL, 1, \"You sent me a NULL data set. Returning NULL.\");\n int apop_varad_var(incol, 0)\n int apop_varad_var(outcol, 0)\n char apop_varad_var(intype, 't')\nAPOP_VAR_ENDHEAD\n if (intype=='t'){\n Apop_stopif(incol >= data->textsize[1], apop_return_data_error(d),\n 0, \"You asked for the text column %i but the \"\n \"data's text has only %zu elements.\", incol, data->textsize[1]);\n } else {\n Apop_stopif((incol == -1) && !data->vector, apop_return_data_error(d),\n 0, \"You asked for the vector of the data set but there is none.\");\n Apop_stopif((incol != -1) && !data->matrix, apop_return_data_error(d),\n 0, \"You asked for the matrix column %i but the matrix is NULL.\", incol);\n Apop_stopif((incol != -1) && (incol >= data->matrix->size2), apop_return_data_error(d),\n 0, \"You asked for the matrix column %i but \"\n \"the matrix has only %zu elements.\", incol, data->matrix->size2);\n }\n if (!data->vector && outcol == -1) //allocate a vector for the user.\n data->vector = gsl_vector_alloc(intype=='t' ? data->textsize[0] : data->matrix->size2);\n if (!data->matrix && outcol >= 0) //allocate a matrxi for the user.\n data->matrix = gsl_matrix_calloc(intype=='t' ? data->textsize[0] : data->matrix->size2, outcol+1);\n apop_data *out;\n dummies_and_factors_core(data, incol, intype, 1, outcol, 'f', &out);\n return out;\n}\n\n\n/** Deprecated. Use \\ref apop_data_to_factors.\n \n Convert a column of text in the text portion of an \\c apop_data set\n into a column of numeric elements, which you can use for a multinomial probit, for example.\n\n\\param d The data set to be modified in place.\n\\param datacol The column in the data set where the numeric factors will be written (-1 means the vector, which I will allocate for you if it is \\c NULL)\n\\param textcol The column in the text that will be converted.\n\nFor example:\n\\code\napop_data *d = apop_query_to_mixed_data(\"mmt\", \"select 1, year, color from data\");\napop_text_to_factors(d, 0, 0);\n\\endcode\nNotice that the query pulled a column of ones for the sake of saving room for the factors.\n\n\\return A table of the factors used in the code. This is an \\c apop_data set with only one column of text.\nAlso, the more element is a reference table of names and column numbers.\n\n\\exception out->error=='d' dimension error.\n*/\napop_data *apop_text_to_factors(apop_data *d, size_t textcol, int datacol){\n Apop_stopif(textcol >= d->textsize[1], apop_return_data_error(d),\n 0, \"You asked for the text element %i but the data's \"\n \"text has only %zu elements.\", datacol, d->textsize[1]);\n if (!d->vector && datacol == -1) //allocate a vector for the user.\n d->vector = gsl_vector_alloc(d->textsize[0]);\n apop_data *out;\n dummies_and_factors_core(d, textcol, 't', 1, datacol, 'f', &out);\n return out;\n}\n\n/** Also known as \\f$R^2\\f$. Let \\f$Y\\f$ be the dependent variable,\n\\f$\\epsilon\\f$ the residual, \\f$n\\f$ the number of data points, and \\f$k\\f$ the number\nof independent vars (including the constant). Returns an \\ref apop_data set with the\nfollowing entries (in the vector element):\n\n\\li \\f$ SST \\equiv \\sum (Y_i - \\bar Y) ^2 \\f$\n\\li \\f$ SSE \\equiv \\sum \\epsilon ^2 \\f$\n\\li \\f$ R^2 \\equiv 1 - {SSE\\over SST} \\f$\n\\li \\f$ R^2_{adj} \\equiv R^2 - {(k-1)\\over (n-k-1)}(1-R^2) \\f$\n\nInternally allocates (and frees) a vector the size of your data set.\n\n\\return A \\f$5 \\times 1\\f$ apop_data table with the following fields:\n\\li \"R squared\"\n\\li \"R squared adj\"\n\\li \"SSE\"\n\\li \"SST\"\n\\li \"SSR\"\n\nIf the output is in \\c sss, use apop_data_get(sss, .rowname=\"SSE\") to get the SSE, and so on for the other items.\n\n\\param m A model. I use the pointer to the data set used for estimation and the info page named \\c \"\". \nThe Predicted page should include observed, expected, and residual columns, which I use to\ngenerate the sums of squared errors and residuals, et cetera. All generalized linear\nmodels produce a page with this name and of this form, as do a host of other models. Nothing \nkeeps you from finding the \\f$R^2\\f$ of, say, a kernel smooth; it is up to you to determine \nwhether such a thing is appropriate to your given models and situation.\n\n\\li apop_estimate(yourdata, apop_ols) does this automatically\n\\li If I don't find a \"\" page, print an error (iff apop_opts.verbose >=0) and return \\c NULL.\n\\li The number of observations equals the number of rows in the Predicted page\n\\li The number of independent variables, needed only for the adjusted \\f$R^2\\f$, is from the\nnumber of columns in the main data set's matrix (i.e. the first page; i.e. the set of\nparameters if this is the \\c parameters output from a model estimation). \n\\li If your data (first page again) has a \\c weights vector, I will find weighted SSE,\nSST, and SSR (and calculate the \\f$R^2\\f$s using those values).\n */\napop_data *apop_estimate_coefficient_of_determination (apop_model *m){\n double sse, sst, rsq, adjustment;\n size_t indep_ct= m->data->matrix->size2 - 1;\n apop_data *out = apop_data_alloc();\n gsl_vector *weights = m->data->weights; //typically NULL.\n apop_data *expected = apop_data_get_page(m->info, \"\");\n Apop_stopif(!expected, return NULL, 0, \"I couldn't find a \\\"\\\" page in your data set. Returning NULL.\\n\");\n size_t obs = expected->matrix->size1;\n Apop_col_tv(expected, \"residual\", v)\n if (!weights)\n gsl_blas_ddot(v, v, &sse);\n else {\n gsl_vector *v_times_w = apop_vector_copy(weights);\n gsl_vector_mul(v_times_w, v);\n gsl_blas_ddot(v_times_w, v, &sse);\n gsl_vector_free(v_times_w);\n }\n gsl_vector *vv = Apop_cv(expected, 0);\n sst = apop_vector_var(vv, m->data->weights) * (vv->size-1);\n rsq = 1. - (sse/sst);\n adjustment = ((obs -1.) /(obs - indep_ct)) * (1.-rsq) ;\n apop_data_add_named_elmt(out, \"R squared\", rsq);\n apop_data_add_named_elmt(out, \"R squared adj\", 1 - adjustment);\n apop_data_add_named_elmt(out, \"SSE\", sse);\n apop_data_add_named_elmt(out, \"SST\", sst);\n apop_data_add_named_elmt(out, \"SSR\", sst - sse);\n return out;\n}\n\n/** \\def apop_estimate_r_squared(in) \n A synonym for \\ref apop_estimate_coefficient_of_determination, q.v. \n \\hideinitializer\n */\n" }, { "path": "apop_settings.c", "content": "/** \\file \n Specifying model characteristics and details of estimation methods. */\n/* Copyright (c) 2008--2009, 2011, 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n#include \"apop_internal.h\"\n\nstatic size_t get_settings_ct(apop_model *model){\n int ct =0;\n if (!model->settings) return 0;\n while (model->settings[ct].name[0] !='\\0') ct++;\n return ct;\n}\n\n//The Dan J Bernstein string hashing algorithm.\n//Could conceivably save a lot of time under certain settings-heavy circumstances.\nstatic unsigned long apop_settings_hash(char *str){\n unsigned long int hash = 5381;\n char c;\n while ((c = *str++)) hash = hash*33 + c;\n return hash;\n}\n\n/* Remove a settings group from a model.\n\nUse \\ref Apop_settings_rm_group. That macro uses this function internally.\n*/\nvoid apop_settings_remove_group(apop_model *m, char *delme){\n if (!m->settings) return;\n int i = 0;\n int ct = get_settings_ct(m);\n unsigned long delme_hash = apop_settings_hash(delme);\n \n while (m->settings[i].name[0] !='\\0'){\n if (m->settings[i].name_hash == delme_hash){\n ((void (*)(void*))m->settings[i].free)(m->settings[i].setting_group);\n for (int j=i+1; j< ct+1; j++) //don't forget the null sentinel.\n m->settings[j-1] = m->settings[j];\n i--;\n }\n i++;\n }\n // apop_assert_void(0, 1, 'c', \"I couldn't find %s in the input model, so nothing was removed.\", delme);\n}\n\n/* Don't use this function. It's what the \\c Apop_model_add_group macro uses internally. Use that. */\nvoid *apop_settings_group_alloc(apop_model *model, char *type, void *free_fn, void *copy_fn, void *the_group){\n if(apop_settings_get_grp(model, type, 'c')) \n apop_settings_remove_group(model, type); \n int ct = get_settings_ct(model);\n model->settings = realloc(model->settings, sizeof(apop_settings_type)*(ct+2)); \n model->settings[ct] = (apop_settings_type) {\n .setting_group = the_group,\n .name_hash = apop_settings_hash(type),\n .free= free_fn, .copy = copy_fn };\n strncpy(model->settings[ct].name, type, 100);\n model->settings[ct+1] = (apop_settings_type) { };\n return model->settings[ct].setting_group;\n}\n\n//need this for the apop_settings_model_group_alloc macro.\napop_model *apop_settings_group_alloc_wm(apop_model *model, char *type, void *free_fn, void *copy_fn, void *the_group){\n apop_settings_group_alloc(model, type, free_fn, copy_fn, the_group);\n return model;\n}\n\n/* This function is used internally by the macro \\ref Apop_settings_get_group. Use that. */\nvoid * apop_settings_get_grp(apop_model *m, char *type, char fail){\n //Used only for finding the non-blank groups.\n Apop_stopif(!m, return NULL, 0, \"you gave me a NULL model as input.\");\n if (!m->settings) return NULL;\n int i;\n unsigned long type_hash = apop_settings_hash(type);\n for (i=0; m->settings[i].name[0] !='\\0'; i++)\n if (type_hash == m->settings[i].name_hash)\n return m->settings[i].setting_group;\n Apop_assert(fail != 'f', \"I couldn't find the settings group %s in the given model.\", type);\n return NULL; //else, just return NULL and let the caller sort it out.\n}\n\n/** Copy a settings group with the given name from the second model to\nthe first (i.e., the arguments are in memcpy order). \n\nYou probably won't need this often---just use \\ref apop_model_copy.\n\n\\param outm The model that will receive a copy of the settings group.\n\\param inm The model that will provide the original.\n\\param copyme The string naming the group. For example, for an \\ref apop_mcmc_settings group, this would be \\c \"apop_mcmc\".\n\n\\exception outm->error=='s' Error copying settings group.\n*/\nvoid apop_settings_copy_group(apop_model *outm, apop_model *inm, char *copyme){\n if (!copyme || !strlen(copyme)) return; //apop_settings_group_alloc takes care of the blank sentinel.\n Apop_stopif(!inm, if (outm) outm->error = 's'; return, 0, \"you asked me to copy the settings of a NULL model.\");\n Apop_stopif(!inm->settings, return, 0, \"The input model (i.e., the second argument to this function) has no settings.\");\n void *g = apop_settings_get_grp(inm, copyme, 'c');\n Apop_stopif(!g, outm->error='s'; return, 0, \"Couldn't find the group you wanted me to copy. Not copying anything; setting outmodel->error='s'.\");\n int i;\n unsigned long type_hash = apop_settings_hash(copyme);\n for (i=0; inm->settings[i].name[0] !='\\0'; i++)//retrieve the index.\n if (type_hash == inm->settings[i].name_hash)\n break;\n void *gnew = (inm->settings[i].copy) \n ? ((void *(*)(void*))inm->settings[i].copy)(g)\n : g;\n apop_settings_group_alloc(outm, copyme, inm->settings[i].free, inm->settings[i].copy, gnew);\n}\n" }, { "path": "apop_sort.m4.c", "content": "/** \\file apop_sort.c\nCopyright (c) 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\nstatic double find_smallest_larger_than(apop_data const *sort_order, double *x){\n //the next column in the sort order is the one that is not NAN, greater than x, but smaller than all other candidate values.\n double candidate_col=-100, candidate_val = INFINITY, v;\n if (sort_order->vector && !isnan(v=gsl_vector_get(sort_order->vector, 0)) \n && v > *x && v < candidate_val){\n candidate_val = v;\n candidate_col = -1;\n }\n if (sort_order->matrix)\n for (int i=0; i< sort_order->matrix->size2; i++)\n if (!isnan(v=gsl_matrix_get(sort_order->matrix, 0, i)) \n && v > *x && v < candidate_val){\n candidate_val = v;\n candidate_col = i;\n }\n if (*sort_order->textsize)\n for (int i=0; i< sort_order->textsize[1]; i++)\n if (apop_opts.nan_string && strcmp(sort_order->text[0][i], apop_opts.nan_string)\n && (v=atof(sort_order->text[0][i])) > *x && v < candidate_val){\n candidate_val = v;\n candidate_col = i+0.5;\n }\n if (sort_order->weights && !isnan(v=gsl_vector_get(sort_order->weights, 0)) \n && v > *x && v < candidate_val){\n candidate_val = v;\n candidate_col = -2;\n }\n if (sort_order->names->rowct)\n if (apop_opts.nan_string && strcmp(*sort_order->names->row, apop_opts.nan_string)\n && (v=atof(*sort_order->names->row)) > *x && v < candidate_val){\n candidate_val = v;\n candidate_col = 0.2;\n }\n\n *x = candidate_val;\n return candidate_col;\n}\n\nstatic void generate_sort_order(apop_data const *data, apop_data const *sort_order, int cols_to_sort_ct, double *so){\n/* the internal rule is that the vector is -1, the weights vector is -2, the names are\n * 0.2, and the text cols are the column+0.5. How's that for arbitrary. */\n if (sort_order) {\n double x = -INFINITY;\n for (int i=0; i< cols_to_sort_ct-1; i++)\n so[i] = find_smallest_larger_than(sort_order, &x);\n } else {\n int ctr=0;\n if (data->vector) so[ctr++] = -1;\n if (data->matrix) for(int i=0; imatrix->size2; i++) so[ctr++] = i;\n if (*data->textsize) for(int i=0; i< data->textsize[1]; i++) so[ctr++] = i+0.5;\n if (data->weights) so[ctr++] = -2;\n if (data->names->rowct) so[ctr++] = 0.2;\n }\n so[cols_to_sort_ct-1] = -100;\n}\n\n#include \n\nstatic int find_min_unsorted(size_t *sorted, size_t height, size_t min){\n while (minnames->row[*da], d->names->row[*db])\n : strcasecmp(d->text[*da][offset], d->text[*db][offset]);\n}\n\nstatic void rearrange(apop_data *data, size_t height, size_t *perm){\n size_t i, start=0;\n size_t *sorted = calloc(height, sizeof(size_t));\n while (1){\n i =\n start = find_min_unsorted(sorted, height, start);\n if (i==-1) break;\n apop_data *first_row_storage = apop_data_copy(Apop_r(data, start));\n sorted[start]++;\n while (perm[i]!=start){\n //copy from perm[i] to i\n apop_data_memcpy(Apop_r(data,i), Apop_r(data, perm[i]));\n sorted[perm[i]]++;\n i = perm[i];\n }\n apop_data_memcpy(Apop_r(data, i), first_row_storage);\n apop_data_free(first_row_storage);\n }\n free(sorted);\n}\n\n/** Sort an \\ref apop_data set on an arbitrary sequence of columns. \n\nThe \\c sort_order set is a one-row data set that should look like the data set being\nsorted. The easiest way to generate it is to use \\ref Apop_r to pull one row of the\ntable, then copy and fill it. For each column you want used in the sort, assign a ranking giving whether the column should be sorted first, second, .... Columns you don't want used in the sorting should be set to \\c NAN. Ties are broken by the earlier element in the default order (see below).\n\nE.g., to sort by the last column of a five-column matrix first, then the next-to-last column, then the next-to-next-to-last, then by the first text column, then by the second text column:\n\n\\code\napop_data *sort_order = apop_data_copy(Apop_r(data, 0));\nsort_order->vector = NULL; //so it will be skipped.\nApop_data_fill(sort_order, NAN, NAN, 3, 2, 1);\napop_text_set(sort_order, 0, 0, \"4\");\napop_text_set(sort_order, 0, 1, \"5\");\napop_data_sort(data, sort_order);\n\\endcode\n\nTo determine which columns are sorted at which step, I use only comparisons, not the actual numeric values. For example, (1, 2, 3) and (-1.32, 0, 27) work identically. For text, I use \\c atof to convert the your text to a number, as in the example above that set text values of \\c \"4\" and \\c \"5\". A blank string, NaN numeric value, or NULL element in the \\ref apop_data set means that column will not be sorted.\n\n\\li Strings are sorted case-insensitively, using \\c strcasecmp. [exercise for the reader: modify the source to use Glib's locale-correct string sorting.]\n\n\\li The setup generates a lexicographic sort using the columns you specify. If you would like a different sort order, such as Euclidian distance to the origin, you can generate a new column expressing your preferred metric, and then sorting on that. See the example below.\n\n\\param data The data set to be sorted. If \\c NULL, this function is a no-op that returns \\c NULL.\n\\param sort_order An \\ref apop_data set describing the order in which columns are used for sorting, as above. If \\c NULL, then sort by the vector, then each matrix column, then text, then weights, then row names.\n\\param inplace If 'n', make a copy, else sort in place. (default: 'y').\n\\param asc If 'a', ascending; if 'd', descending. This is applied to all columns; column-by-column application is to do. (default: 'a').\n\\param col_order For internal use only. In your call, it should be \\c NULL; you can leave this off your function call entirely and the \\ref designated syntax will takes care of it for you.\n\n\\return A pointer to the sorted data set. If inplace=='y' (the default), then this is the same as the input set.\n\n\nA few examples:\n\n\\include \"sort_example.c\"\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_data *apop_data_sort(apop_data *data, apop_data *sort_order, char asc, char inplace, double *col_order){\n apop_data * apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, 1, \"You gave me NULL data to sort. Returning NULL\");\n apop_data * apop_varad_var(sort_order, NULL);\n char apop_varad_var(inplace, 'y');\n char apop_varad_var(asc, 'a');\n double * apop_varad_var(col_order, NULL);\nAPOP_VAR_ENDHEAD\n if (!data) return NULL;\n\n apop_data *out = inplace=='n' ? apop_data_copy(data) : data;\n\n apop_data *xx = sort_order ? sort_order : out;\n Get_vmsizes(xx); //firstcol, msize2\n int cols_to_sort_ct = msize2 - firstcol +1 + !!(xx->weights) + xx->textsize[1] + !!xx->names->rowct;\n double so[cols_to_sort_ct];\n if (!col_order){\n generate_sort_order(out, sort_order, cols_to_sort_ct, so);\n col_order = so;\n }\n\n bool is_text = ((int)*col_order != *col_order);\n bool is_name = (*col_order == 0.2);\n\n gsl_vector_view c;\n gsl_vector *cc = NULL;\n if (!is_text && *col_order>=0){\n c = gsl_matrix_column(out->matrix, *col_order);\n cc = &c.vector;\n }\n gsl_vector *thiscol = cc ? cc\n : (*col_order==-2) ? out->weights\n : (*col_order==-1) ? out->vector\n : NULL;\n\n size_t height = thiscol ? thiscol->size\n : is_name ? out->names->rowct\n : *out->textsize;\n\n gsl_permutation *p = gsl_permutation_alloc(height);\n if (!is_text) gsl_sort_vector_index (p, thiscol);\n else {\n gsl_permutation_init(p);\n d = out;\n offset = is_name ? -1 : *col_order-0.5; \n qsort(p->data, height, sizeof(size_t), compare_strings);\n }\n\n size_t *perm = p->data;\n if (asc=='d' || asc=='D') //reverse the perm matrix.\n for (size_t j=0; j< height/2; j++){\n double t = perm[j];\n perm[j] = perm[height-1-j];\n perm[height-1-j] = t;\n }\n rearrange(out, height, perm);\n gsl_permutation_free(p);\n if (col_order[1] == -100) return out;\n\n /*Second pass:\n find blocks where all are of the same value.\n After you pass a block of size > 1 row where all vals in this col are identical,\n sort that block, using the rest of the sort order. */\n int bottom=0;\n if (!is_text){\n double last_val = gsl_vector_get(thiscol, 0);\n for (int i=1; i< height+1; i++){\n double this_val=0;\n if ((i==height || (this_val=gsl_vector_get(thiscol, i)) != last_val) \n && bottom != i-1){\n apop_data_sort_base(Apop_rs(out, bottom, i-bottom), sort_order, 'a', 'y', col_order+1);\n }\n if (last_val != this_val) bottom = i;\n last_val = this_val;\n }\n } else {\n char *last_val = strdup(is_name ? out->names->row[0] : out->text[0][(int)(*col_order-0.5)]);\n for (int i=1; i< height+1; i++){\n char *this_val = i==height ? NULL : is_name ? out->names->row[i] : out->text[i][(int)(*col_order-0.5)];\n if ((i==height || strcasecmp(this_val, last_val)) \n && bottom != i-1){\n apop_data_sort_base(Apop_rs(out, bottom, i-bottom), sort_order, 'a', 'y', col_order+1);\n }\n if (this_val && strcmp(last_val, this_val)) bottom = i;\n free(last_val);\n last_val = this_val ? strdup(this_val) : NULL;\n }\n free(last_val);\n }\n return out;\n}\n" }, { "path": "apop_stats.m4.c", "content": "/** \\file apop_stats.c\tBasic moments and some distributions. */\n/* Copyright (c) 2006--2007, 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n#include \n\n#define Check_vw \\\n Apop_stopif(!v, return GSL_NAN, 0, \"data vector is NULL. Returning NaN.\\n\"); \\\n Apop_stopif(!v->size, return GSL_NAN, 0, \"data vector has size 0. Returning NaN.\\n\"); \\\n Apop_stopif(weights && weights->size != v->size, return GSL_NAN, 0, \"data vector has size %zu; weighting vector has size %zu. Returning NaN.\\n\", v->size, weights->size);\n\n/** Returns the sum of the data in the given vector.\n*/\nlong double apop_vector_sum(const gsl_vector *in){\n Apop_stopif(!in, return 0, 1, \"You just asked me to sum a NULL. Returning zero.\");\n long double out = 0;\n for (size_t i=0; i< in->size; i++)\n out += gsl_vector_get(in, i);\n\treturn out; \n}\n\n/** \\def apop_sum(in)\n An alias for \\ref apop_vector_sum. Returns the sum of the data in the given vector.\n*/\n\n/** \\def apop_mean(v)\n Returns the mean of the elements of the vector \\c v.\n\n\\param v A \\ref gsl_vector.\n*/\n\n/** \\def apop_var(in)\nAn alias for \\ref apop_vector_var.\nReturns the variance of the data in the given vector.\n*/\n\n/** Returns an unbiased estimate of the sample skew of the data in the given vector.\n*/\ndouble apop_vector_skew(const gsl_vector *in){\n\treturn apop_vector_skew_pop(in) * gsl_pow_2(in->size)/((in->size -1.)*(in->size -2.)); }\n\n/** Returns the sample fourth central moment of the data in the given\nvector. Corrections are made to produce an unbiased result as per Appendix M (PDF) of Modeling\nwith data.\n\n\\li This is an estimate of the fourth central moment without normalization. The kurtosis\n of a \\f${\\cal N}(0,1)\\f$ is \\f$3 \\sigma^4\\f$, not three, one, or zero.\n\\see \\ref apop_vector_kurtosis_pop\n*/\ndouble apop_vector_kurtosis(const gsl_vector *in){\n size_t n = in->size;\n long double coeff0= n*n/(gsl_pow_3(n-1)*(gsl_pow_2(n)-3*n+3));\n long double coeff1= n*gsl_pow_2(n-1)+ (6*n-9);\n long double coeff2= n*(6*n-9);\n return coeff0 *(coeff1 * apop_vector_kurtosis_pop(in) - coeff2 * gsl_pow_2(apop_vector_var(in)*(n-1.)/n));\n}\n\nstatic double wskewkurt(const gsl_vector *v, const gsl_vector *w, const int exponent, const char *fn_name){\n long double wsum = 0, sumcu = 0, vv, ww, mu;\n //Using the E(x - \\bar x)^3 form, which is lazy.\n mu = apop_vector_mean(v, w);\n for (size_t i=0; i< w->size; i++){\n vv = gsl_vector_get(v,i);\n ww = gsl_vector_get(w,i);\n sumcu+= ww * gsl_pow_int(vv - mu, exponent); \n wsum += ww; \n }\n double len = wsum < 1.1 ? w->size : wsum;\n return sumcu/len;\n}\n\n/** Returns the population skew \\f$(\\sum_i (x_i - \\mu)^3/n))\\f$ of the data in the given vector. Observations may be weighted.\n\n\\param v The data vector\n\\param weights The weight vector. Default: equal weights for all observations.\n\\return The weighted skew.\n \n\\li Some people like to normalize the skew by dividing by (variance)\\f$^{3/2}\\f$; that's not done here, so you'll have to do so separately if need be.\n\n\\li Apophenia tries to be smart about reading the weights. If weights\nsum to one, then the system uses \\c w->size as the number of elements,\nand returns the usual sum over \\f$n-1\\f$. If weights > 1, then the\nsystem uses the total weights as \\f$n\\f$. Thus, you can use the weights\nas standard weightings or to represent elements that appear repeatedly.\n*/\nAPOP_VAR_HEAD double apop_vector_skew_pop(gsl_vector const *v, gsl_vector const *weights){\n gsl_vector const * apop_varad_var(v, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\n Check_vw\nAPOP_VAR_ENDHEAD\n if (weights) return wskewkurt(v, weights, 3, \"apop_vector_weighted_skew\");\n\n //This code is cut/pasted/modified from the GSL. \n //I reimplement the skew calculation here without the division by var^3/2 that the GSL does. \n size_t n = v->size;\n long double avg = 0;\n long double mean = apop_vector_mean(v);\n for (size_t i = 0; i < n; i++) {\n const long double x = gsl_vector_get(v, i) - mean; \n avg += (gsl_pow_3(x) - avg)/(i + 1);\n } \n return avg;\n}\n\n/** Returns the population fourth central moment [\\f$\\sum_i (x_i - \\mu)^4/n)\\f$] of the data in\nthe given vector, with an optional weighting.\n\n\\param v The data vector\n\\param weights The weight vector. If NULL, assume equal weights.\n\\return The weighted kurtosis.\n \n \\li Some people like to normalize the fourth central moment by dividing by variance\nsquared, or by subtracting three; those things are not done here, so you'll have to\ndo them separately if desired.\n \\li This function uses the \\ref designated syntax for inputs.\n\\see \\ref apop_vector_kurtosis for the unbiased sample version.\n*/\nAPOP_VAR_HEAD double apop_vector_kurtosis_pop(gsl_vector const *v, gsl_vector const *weights){\n gsl_vector const * apop_varad_var(v, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\n Check_vw\nAPOP_VAR_ENDHEAD\n if (weights) return wskewkurt(v, weights, 4, \"apop_vector_weighted_kurtosis\");\n\n //This code is cut/pasted/modified from the GSL. \n //I reimplement the kurtosis calculation here without the division by var^2 that the GSL does. \n size_t n = v->size;\n long double avg = 0;\n long double mean = apop_vector_mean(v);\n for (size_t i = 0; i < n; i++) {\n const long double x = gsl_vector_get(v, i) - mean; \n avg += (gsl_pow_4(x) - avg)/(i + 1);\n } \n return avg;\n}\n\n/** Returns the variance of the data in the given vector, given that you've already calculated the mean.\n\\param in\tthe vector in question\n\\param mean\tthe mean, which you've already calculated using \\ref apop_vector_mean.\n\\see apop_vector_var\n*/\ndouble apop_vector_var_m(const gsl_vector *in, const double mean){\n\treturn gsl_stats_variance_m(in->data,in->stride, in->size, mean); }\n\n/** Returns the correlation coefficient of two vectors:\n\\f$ {\\hbox{cov}(a,b)\\over \\sqrt{\\hbox{var}(a)} \\sqrt{\\hbox{var}(b)}}.\\f$\n\nAn example\n\\code \ngsl_matrix *m = apop_text_to_data(\"indata\")->matrix;\nprintf(\"The correlation coefficient between rows two \"\n \"and three is %g.\\n\", apop_vector_correlation(Apop_mrv(m, 2), Apop_mrv(m, 3)));\n\\endcode \n\n\\param ina, inb Two vectors of equal length (no default, must not be NULL)\n\\param weights Replicate weights for the observations. (default: equal weights for all observations)\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD double apop_vector_correlation(const gsl_vector *ina, const gsl_vector *inb, const gsl_vector *weights){\n gsl_vector const * apop_varad_var(ina, NULL);\n gsl_vector const * apop_varad_var(inb, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\nAPOP_VAR_ENDHEAD\n\treturn apop_vector_cov(ina, inb, weights) \n / sqrt(apop_vector_var(ina, weights) * apop_vector_var(inb, weights)); }\n\n\n/** Returns the distance between two vectors, where distance is defined\n based on the third (optional) parameter:\n\n - 'e' (the default): scalar distance (standard Euclidean metric) between two vectors. \\f$\\sqrt{\\sum_i{(a_i - b_i)^2}},\\f$\nwhere \\f$i\\f$ iterates over dimensions.\n - 'm' Returns the Manhattan metric distance between two vectors: \\f$\\sum_i{|a_i - b_i|},\\f$\nwhere \\f$i\\f$ iterates over dimensions.\n - 'd' The discrete norm: if \\f$a = b\\f$, return zero, else return one.\n - 's' The sup norm: find the dimension where \\f$|a_i - b_i|\\f$ is largest, return the distance along that one dimension.\n - 'l' or 'L' The \\f$L_p\\f$ norm, \\f$\\left(\\sum_i{|a_i - b_i|^2}\\right)^{1/p}\\f$. The value of \\f$p\\f$ is set by the fourth (optional) argument.\n\n \\param ina First vector (No default, must not be \\c NULL)\n \\param inb Second vector (Default = zero)\n \\param metric The type of metric, as above.\n \\param norm If you are using an \\f$L_p\\f$ norm, this is \\f$p\\f$. Must be strictly greater than zero. (default = 2)\n\n\\li The defaults are such that\n \\code\n apop_vector_distance(v);\n apop_vector_distance(v, .metric = 's');\n apop_vector_distance(v, .metric = 'm');\n \\endcode\ngives you the standard Euclidean length of \\c v, its longest element, and its sum.\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\include test_distances.c\n*/\nAPOP_VAR_HEAD double apop_vector_distance(const gsl_vector *ina, const gsl_vector *inb, const char metric, const double norm){\n static threadlocal gsl_vector *zero = NULL;\n const gsl_vector * apop_varad_var(ina, NULL);\n Apop_stopif(!ina, return NAN, 1, \"The first vector is NULL. Returning NAN\");\n const gsl_vector * apop_varad_var(inb, NULL);\n if (!inb){\n if (!zero || zero->size !=ina->size){\n if (zero) gsl_vector_free(zero);\n zero = gsl_vector_calloc(ina->size);\n }\n inb = zero;\n }\n const char apop_varad_var(metric, 'e');\n const double apop_varad_var(norm, 2);\nAPOP_VAR_ENDHEAD\n Apop_stopif(ina->size != inb->size, return GSL_NAN, 0, \"I need equal-sized vectors, but \"\n \"you sent a vector of size %zu and a vector of size %zu. Returning NaN.\", ina->size, inb->size);\n double dist = 0;\n if (metric == 'e' || metric == 'E'){\n for (size_t i=0; i< ina->size; i++)\n dist += gsl_pow_2(gsl_vector_get(ina, i) - gsl_vector_get(inb, i));\n return sqrt(dist); \n }\n if (metric == 'm' || metric == 'M'){ //redundant with vector_grid_distance, below.\n for (size_t i=0; i< ina->size; i++) \n dist += fabs(gsl_vector_get(ina, i) - gsl_vector_get(inb, i));\n return dist; \n }\n if (metric == 'd' || metric == 'D'){\n for (size_t i=0; i< ina->size; i++) \n if (gsl_vector_get(ina, i) != gsl_vector_get(inb, i))\n return 1;\n return 0;\n }\n if (metric == 's' || metric == 'S'){\n for (size_t i=0; i< ina->size; i++) \n dist = GSL_MAX(dist, fabs(gsl_vector_get(ina, i) - gsl_vector_get(inb, i)));\n return dist;\n }\n if (metric == 'l' || metric == 'L'){\n for (size_t i=0; i< ina->size; i++)\n dist += pow(fabs(gsl_vector_get(ina, i) - gsl_vector_get(inb, i)), norm);\n return pow(dist, 1./norm); \n }\n Apop_stopif(1, return NAN, 1, \"I couldn't find the metric type you gave, %c, in my list of supported types. Returning NaN\", metric);\n}\n\n/** This function will normalize a vector, either such that it has mean\nzero and variance one, or ranges between zero and one, or sums to one.\n\n\\param in\tA \\c gsl_vector with the un-normalized data. \\c NULL\ninput gives \\c NULL output. (No default)\n\n\\param out \tIf normalizing in place, \\c NULL.\nIf not, the address of a gsl_vector*. Do not allocate. (default = \\c NULL.)\n\n\\param normalization_type \n\\c 'p': normalized vector will sum to one. E.g., start with a set of observations in bins, end with the percentage of observations in each bin. (the default)
\n\\c 'r': normalized vector will range between zero and one. Replace each X with (X-min) / (max - min).
\n\\c 's': normalized vector will have mean zero and (sample) variance one. Replace\neach X with \\f$(X-\\mu) / \\sigma\\f$, where \\f$\\sigma\\f$ is the sample\nstandard deviation.
\n\\c 'm': normalize to mean zero: Replace each X with \\f$(X-\\mu)\\f$
\n\n\\b Example \n\\code\n\\endcode\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD void apop_vector_normalize(gsl_vector *in, gsl_vector **out, const char normalization_type){\n gsl_vector * apop_varad_var(in, NULL);\n Apop_stopif(!in, return, 1, \"Input vector is NULL. Doing nothing.\");\n gsl_vector ** apop_varad_var(out, NULL);\n const char apop_varad_var(normalization_type, 'p');\nAPOP_VAR_END_HEAD\n double mu, min, max;\n\tif (!out) out = ∈\n\telse {\n\t\t*out = gsl_vector_alloc (in->size);\n\t\tgsl_vector_memcpy(*out, in);\n\t}\n\tif (normalization_type == 's'){\n\t\tmu = apop_vector_mean(in);\n Apop_stopif(!isfinite(mu), return, 0, \"normalization failed: the mean of the vector is not finite.\");\n\t\tgsl_vector_add_constant(*out, -mu);\n double scaling = 1./(sqrt(apop_vector_var_m(*out, 0)));\n Apop_stopif(!isfinite(scaling), return, 0, \"normalization failed: 1/(std error) of the vector is not finite.\");\n\t\tgsl_vector_scale(*out, scaling);\n\t} \n\telse if (normalization_type == 'r'){\n gsl_vector_minmax(in, &min, &max);\n\t\tgsl_vector_add_constant(*out, -min);\n\t\tgsl_vector_scale(*out, 1/(max-min));\t\n\n\t}\n\telse if (normalization_type == 'p'){\n\t\tlong double sum\t= apop_sum(in);\n Apop_stopif(!sum, return, 0, \"the vector sums to zero, so I can't normalize it to sum to one.\");\n\t\tgsl_vector_scale(*out, 1/sum);\t\n\t}\n\telse if (normalization_type == 'm'){\n\t\tmu = apop_vector_mean(in);\n Apop_stopif(!isfinite(mu), return, 0, \"normalization failed: the mean of the vector is not finite.\");\n\t\tgsl_vector_add_constant(*out, -mu);\n\t}\n}\n\n/** Returns the sum of the elements of a matrix. Occasionally convenient.\n\n\\param m\tthe matrix to be summed. \n*/\nlong double apop_matrix_sum(const gsl_matrix *m){\n Apop_stopif(!m, return 0, 1, \"You just asked me to sum a NULL. Returning zero.\");\n long double\tsum\t= 0;\n\tfor (size_t j=0; j< m->size1; j++)\n\t\tfor (size_t i=0; i< m->size2; i++)\n\t\t\tsum += gsl_matrix_get(m, j, i);\n\treturn sum;\n}\n\n/** Returns the mean of all elements of a matrix.\n\n\\param data\tThe matrix to be averaged. If \\c NULL, return zero.\n\\return The mean of all cells of the matrix.\n*/\ndouble apop_matrix_mean(const gsl_matrix *data){\n if (!data) return 0;\n long double avg = 0;\n int cnt = 0;\n for(size_t i=0; i < data->size1; i++)\n for(size_t j=0; j < data->size2; j++){\n double x = gsl_matrix_get(data, i,j);\n long double ratio = cnt/(cnt+1.0);\n cnt++;\n avg*= ratio;\n avg+= x/cnt;\n }\n\treturn avg;\n}\n\n/** Returns the mean and population variance of all elements of a matrix.\n \n\\li If \\c NULL, return \\f$\\mu=0, \\sigma^2=NaN\\f$.\n\\li Gives the population variance (sum of squares divided by \\f$N\\f$). \nIf you want sample variance, multiply the result by \\f$N/(N-1)\\f$:\n\\code\ndouble mu, var;\napop_data *data= apop_query_to_data(\"select * from indata\");\napop_matrix_mean_and_var(data->matrix, &mu, &var);\nvar *= (data->size1*data->size2)/(data->size1*data->size2-1.0);\n\\endcode\n\n\\param data\tthe matrix to be averaged. \n\\param\tmean\twhere to put the mean to be calculated.\n\\param\tvar\twhere to put the variance to be calculated.\n*/\nvoid apop_matrix_mean_and_var(const gsl_matrix *data, double *mean, double *var){\n if (!data) {*mean=0; *var=GSL_NAN; return;}\n long double avg = 0,\n avg2 = 0;\n size_t cnt= 0;\n long double x, ratio;\n for(size_t i=0; i < data->size1; i++)\n for(size_t j=0; j < data->size2; j++){\n x = gsl_matrix_get(data, i,j);\n ratio = cnt/(cnt+1.0);\n cnt ++;\n avg *= ratio;\n avg2 *= ratio;\n avg += x/(cnt +0.0);\n avg2 += gsl_pow_2(x)/(cnt +0.0);\n }\n\t*mean = avg;\n *var = avg2 - gsl_pow_2(avg); //E[x^2] - E^2[x]\n}\n\n/** Put summary information about the columns of a table (mean, std dev, variance, min, median, max) in a table.\n\n\\param indata The table to be summarized. An \\ref apop_data structure. May have a weights element.\n\\return An \\ref apop_data structure with one row for each column in the original\n table, and a column for each summary statistic.\n\\exception out->error='a' Allocation error.\n\n\\li This function gives more columns than you probably want; use \\ref apop_data_prune_columns to pick the ones you want to see.\n\n\\li See apop_data_prune_columns for an example.\n*/\napop_data * apop_data_summarize(apop_data *indata){\n Apop_stopif(!indata, return NULL, 0, \"You sent me a NULL apop_data set. Returning NULL.\");\n Apop_stopif(!indata->matrix, return NULL, 0, \"You sent me an apop_data set with a NULL matrix. Returning NULL.\");\n apop_data *out = apop_data_alloc(indata->matrix->size2, 6);\n double mean, var;\n char rowname[10000]; //crashes on more than 10^9995 columns.\n\tapop_name_add(out->names, \"mean\", 'c');\n\tapop_name_add(out->names, \"std dev\", 'c');\n\tapop_name_add(out->names, \"variance\", 'c');\n\tapop_name_add(out->names, \"min\", 'c');\n\tapop_name_add(out->names, \"median\", 'c');\n\tapop_name_add(out->names, \"max\", 'c');\n\tif (indata->names !=NULL){\n apop_name_stack(out->names,indata->names, 'r', 'c');\n if (indata->names->title && strlen(indata->names->title)){\n char *title;\n Asprintf(&title, \"summary for %s\", indata->names->title);\n apop_name_add(out->names, title, 'h');\n free(title);\n }\n }\n\telse\n\t\tfor (size_t i=0; i< indata->matrix->size2; i++){\n\t\t\tsprintf(rowname, \"col %zu\", i);\n\t\t\tapop_name_add(out->names, rowname, 'r');\n\t\t}\n\tfor (size_t i=0; i< indata->matrix->size2; i++){\n gsl_vector *v = Apop_cv(indata, i);\n if (!indata->weights){\n mean = apop_vector_mean(v);\n var = apop_vector_var_m(v, mean);\n } else {\n mean = apop_vector_mean(v, indata->weights);\n var = apop_vector_var(v, indata->weights);\n } \n double *pctiles = apop_vector_percentiles(v);\n\t\tgsl_matrix_set(out->matrix, i, 0, mean);\n\t\tgsl_matrix_set(out->matrix, i, 1, sqrt(var));\n\t\tgsl_matrix_set(out->matrix, i, 2, var);\n\t\tgsl_matrix_set(out->matrix, i, 3, pctiles[0]);\n\t\tgsl_matrix_set(out->matrix, i, 4, pctiles[50]);\n\t\tgsl_matrix_set(out->matrix, i, 5, pctiles[100]);\n free(pctiles);\n\t}\n\treturn out;\n}\n\n/** Returns an array of size 101, where \\c returned_vector[95] gives the value of the\n95th percentile, for example. \\c Returned_vector[100] is always the maximum value,\nand \\c returned_vector[0] is always the min (regardless of rounding rule).\n\n \\param data\tA \\c gsl_vector with the data. (No default, must not be \\c NULL.)\n \\param rounding Either be \\c 'u', \\c 'd', or \\c 'a'. Unless your data is\nexactly a multiple of 101, some percentiles will be ambiguous. If \\c 'u', then round\nup (use the next highest value); if \\c 'd', round down to the next lowest value; if \\c\n'a', take the mean of the two nearest points. (Default = \\c 'd'.)\n\n\\li If the rounding method is \\c 'u' or \\c 'a', then you can say \"5% or more of\nthe sample is below returned_vector[5]\"; if \\c 'd' or \\c 'a', then you can say \"5%\nor more of the sample is above returned_vector[5]\".\n\\li You may eventually want to \\c free() the array returned by this function.\n\\li This function uses the \\ref designated syntax for inputs.\n*/ \nAPOP_VAR_HEAD double * apop_vector_percentiles(gsl_vector *data, char rounding){\n gsl_vector *apop_varad_var(data, NULL);\n Apop_stopif(!data, return NULL, 0, \"You gave me NULL data.\");\n char apop_varad_var(rounding, 'd');\nAPOP_VAR_ENDHEAD\n gsl_vector *sorted\t= gsl_vector_alloc(data->size);\n double *pctiles = malloc(sizeof(double) * 101);\n\tgsl_vector_memcpy(sorted,data);\n\tgsl_sort_vector(sorted);\n\tfor(int i=0; i<101; i++){\n\t\tint index = i*(data->size-1)/100.0;\n\t\tif (rounding == 'u' && index != i*(data->size-1)/100.0)\n\t\t\tindex ++; //index was rounded down, but should be rounded up.\n\t\tif (rounding == 'a' && index != i*(data->size-1)/100.0)\n pctiles[i]\t= (gsl_vector_get(sorted, index)+gsl_vector_get(sorted, index+1))/2.;\n else pctiles[i]\t= gsl_vector_get(sorted, index);\n\t}\n\tgsl_vector_free(sorted);\n\treturn pctiles;\n}\n\n/** Find the mean, weighted or unweighted. \n\n\\param v The data vector\n\\param weights The weight vector. Default: assume equal weights.\n\\return The weighted mean \n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD double apop_vector_mean(gsl_vector const *v, gsl_vector const *weights){\n gsl_vector const * apop_varad_var(v, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\n Check_vw\nAPOP_VAR_END_HEAD\n if (!weights) return gsl_stats_mean(v->data, v->stride, v->size);\n long double sum = 0, wsum = 0;\n for (size_t i=0; i< weights->size; i++){\n sum += gsl_vector_get(weights, i) * gsl_vector_get(v,i); \n wsum += gsl_vector_get(weights, i); \n }\n return sum/wsum;\n}\n\n/** Find the sample variance of a vector, weighted or unweighted.\n\n\\param v The data vector\n\\param weights The weight vector. If NULL (the default), assume equal weights.\n\\return The weighted sample variance. \n\n \\li This uses (n-1) in the denominator of the sum; i.e., it corrects for the bias\nintroduced by using \\f$\\bar x\\f$ instead of \\f$\\mu\\f$.\n \\li Multiply the output by (n-1)/n if you need population variance.\n \\li Apophenia tries to be smart about reading the weights. If weights\nsum to one, then the system uses \\c w->size as the number of elements,\nand returns the usual sum over \\f$n-1\\f$. If weights > 1, then the\nsystem uses the total weights as \\f$n\\f$. Thus, you can use the weights\nas standard weightings or to represent elements that appear repeatedly.\n \\li This function uses the \\ref designated syntax for inputs.\n\\see apop_vector_var_m for the case where you already have the vector's mean.\n*/\nAPOP_VAR_HEAD double apop_vector_var(gsl_vector const *v, gsl_vector const *weights){\n gsl_vector const * apop_varad_var(v, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\n Check_vw\nAPOP_VAR_END_HEAD\n if (!weights) return gsl_stats_variance(v->data, v->stride, v->size);\n //Using the E(x^2) - E^2(x) form.\n long double sum = 0, wsum = 0, sumsq = 0, vv, ww;\n for (size_t i=0; i< weights->size; i++){\n vv = gsl_vector_get(v, i);\n ww = gsl_vector_get(weights, i);\n sum += ww * vv;\n sumsq += ww * gsl_pow_2(vv); \n wsum += ww; \n }\n double len = (wsum < 1.1 ? weights->size : wsum);\n return (sumsq/len - gsl_pow_2(sum/len)) * len/(len -1.);\n}\n\n/** Find the sample covariance of a pair of vectors, with an optional weighting. This only\nmakes sense if the weightings are identical, so the function takes only one weighting vector for both.\n\n\\param v1, v2 The data vectors (no default; must not be \\c NULL)\n\\param weights The weight vector. (default equal weights for all elements)\n\\return The sample covariance\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD double apop_vector_cov(const gsl_vector *v1, const gsl_vector *v2, const gsl_vector *weights){\n gsl_vector const * apop_varad_var(v1, NULL);\n gsl_vector const * apop_varad_var(v2, NULL);\n gsl_vector const * apop_varad_var(weights, NULL);\n Apop_stopif(!v1, return GSL_NAN, 0, \"first data vector is NULL. Returning NaN.\");\n Apop_stopif(!v2, return GSL_NAN, 0, \"second data vector is NULL. Returning NaN.\");\n Apop_stopif(!v1->size, return GSL_NAN, 0, \"first data vector has size 0. Returning NaN.\");\n Apop_stopif(!v2->size, return GSL_NAN, 0, \"second data vector has size 0. Returning NaN.\");\n Apop_stopif(v1->size!= v2->size, return GSL_NAN, 0, \"data vectors have sizes %zu and %zu. Returning NaN.\", v1->size, v2->size);\n Apop_stopif(weights && ((weights->size != v1->size) || (weights->size != v2->size)), return GSL_NAN, 0, \"data vectors have sizes %zu and %zu; weighting vector has size %zu. Returning NaN.\", v1->size, v2->size, weights->size);\n\nAPOP_VAR_ENDHEAD\n if (!weights) return gsl_stats_covariance(v1->data, v1->stride, v2->data, v2->stride, v2->size);\n long double sum1 = 0, sum2 = 0, wsum = 0, sumsq = 0, vv1, vv2, ww;\n //Using the E(x^2) - E^2(x) form.\n for (size_t i=0; i< weights->size; i++){\n vv1 = gsl_vector_get(v1,i);\n vv2 = gsl_vector_get(v2,i);\n ww = gsl_vector_get(weights,i);\n sum1 += ww * vv1;\n sum2 += ww * vv2;\n sumsq+= ww * vv1 * vv2;\n wsum += ww; \n }\n double len = (wsum < 1.1 ? weights->size : wsum);\n return (sumsq/len - sum1*sum2/gsl_pow_2(len)) *(len/(len-1));\n}\n\n/** Returns the sample variance/covariance matrix relating each column of the matrix to each other column.\n\n\\param in An \\ref apop_data set. If the weights vector is set, I'll take it into account.\n\n\\li This is the sample covariance---dividing by \\f$n-1\\f$, not \\f$n\\f$. If you need the population variance, use \n\\code\napop_data *popcov = apop_data_covariance(indata);\nint size=indata->matrix->size1;\ngsl_matrix_scale(popcov->matrix, size/(size-1.));\n\\endcode\n\n\\return Returns an \\ref apop_data set the variance/covariance matrix. \n\\exception out->error='a' Allocation error.\n*/\napop_data *apop_data_covariance(const apop_data *in){\n Apop_stopif(!in, return NULL, 1, \"You sent me a NULL apop_data set. Returning NULL.\");\n Apop_stopif(!in->matrix, return NULL, 1, \"You sent me an apop_data set with a NULL matrix. Returning NULL.\");\n apop_data *out = apop_data_alloc(in->matrix->size2, in->matrix->size2);\n Apop_stopif(out->error, return out, 0, \"allocation error.\");\n for (size_t i=0; i < in->matrix->size2; i++){\n for (size_t j=i; j < in->matrix->size2; j++){\n double var = apop_vector_cov(Apop_cv(in, i), Apop_cv(in, j), in->weights);\n gsl_matrix_set(out->matrix, i,j, var);\n if (i!=j) gsl_matrix_set(out->matrix, j,i, var);\n }\n }\n apop_name_stack(out->names, in->names, 'c');\n apop_name_stack(out->names, in->names, 'r', 'c');\n return out;\n}\n\n/** Returns the matrix of correlation coefficients \\f$(\\sigma^2_{xy}/(\\sigma_x\\sigma_y))\\f$ relating each column with each other.\n\n\\param in \tA data matrix: rows are observations, columns are variables. If you give me a weights vector, I'll use it.\n\n\\return Returns the square variance/covariance matrix with dimensions equal to the number of input columns.\n\\exception out->error='a' Allocation error.\n*/\napop_data *apop_data_correlation(const apop_data *in){\n apop_data *out = apop_data_covariance(in);\n if (!out) return NULL;\n for(size_t i=0; i< in->matrix->size2; i++){\n double std_dev = sqrt(apop_vector_var(Apop_cv(in, i), in->weights));\n gsl_vector_scale(Apop_cv(out, i), 1.0/std_dev);\n gsl_vector_scale(Apop_rv(out, i), 1.0/std_dev);\n }\n return out;\n}\n\n\n/** Given a vector representing a probability distribution of observations, calculate the entropy, \\f$\\sum_i -\\ln(v_i)v_i\\f$.\n\n\\li You may input a vector giving frequencies (normalized to sum to one) or counts (arbitrary sum).\n\n\\li The entropy of a data set depends only on the frequency with which elements are\nobserved, not the value of the elements themselves. The \\ref apop_data_pmf_compress\nfunction will reduce an input \\ref apop_data set to one weighted line per observation, and\nthe weights would determine the entropy:\n\n\\code\napop_data *data = apop_text_to_data(\"indata\");\napop_data_pmf_compress(data);\ndata_entropy = apop_vector_entropy(d->weights);\n\\endcode\n\n\\li The entropy is calculated using natural logs. To convert to base 2, divide by \\f$\\ln(2)\\f$; see the example.\n\n\\li The entropy of an empty data set (\\c NULL or a total weight of zero) is zero. Print a warning when given \\c NULL\n input and apop_opts.verbose >=1.\n\n\\li If the input vector has negative elements, return \\c NaN; print a warning when apop_opts.verbose >= 0.\n\nSample code:\n\\include entropy_vector.c\n*/\nlong double apop_vector_entropy(gsl_vector *in){\n Apop_stopif(!in, return 0, 1, \"Entropy of a NULL vector ≡ 0\");\n Apop_stopif(!in->size, return 0, 1, \"Entropy of a zero-length vector ≡ 0\");//can't happen.\n\n //User may or may not have normalized in, so scale everything by the sum.\n long double sum = apop_vector_sum(in);\n Apop_stopif(sum<0, return NAN, 0, \"Vector sums to a negative value (%Lg). Returning NaN.\\n\", sum);\n if (!sum) return 0;\n\n long double out=0;\n for (int i=0; i< in->size; i++){\n double val = gsl_vector_get(in, i)/sum;\n Apop_stopif(val<0, return NAN, 0, \"negative value (%g) in vector position %i. Returning NaN.\\n\", val, i);\n if (!val) continue;\n out -= logl(val)*val;\n }\n return out;\n}\n\nstatic long double norment(apop_model *m){\n double sigma_sq = gsl_pow_2(apop_data_get(m->parameters, 1));\n return (log(2*M_PI*sigma_sq) +1)/2.;\n}\n\ndouble get_ll(apop_data *d, void *m){ return apop_log_likelihood(d, m); }\n\n\n/** Calculate the entropy of a model: \\f$\\int -\\ln(p(x))p(x)dx\\f$, which is the expected\n value of \\f$-\\ln(p(x))\\f$.\n\nThe default method is to make draws using \\ref apop_model_draws, then\nevaluate the log likelihood at those points using the model's \\c log_likelihood method.\n\nThere are a number of routines for specific models, inlcuding the \\ref apop_normal and \\ref apop_pmf models.\n\n\\li If you want the entropy of a data set, see \\ref apop_vector_entropy.\n\\li The entropy is calculated using natural logs. If you prefer base-2 logs, just divide by \\f$\\ln(2)\\f$: apop_model_entropy(my_model)/log(2).\n\n\\param in A parameterized \\ref apop_model. That is, you have already used \\ref apop_estimate or \\ref apop_model_set_parameters to estimate/set the model parameters.\n\\param draws If using the default method of making random draws, how many random draws to make (default=1,000)\n\nSample code:\n\\include entropy_model.c\n*/\nAPOP_VAR_HEAD long double apop_model_entropy(apop_model *in, int draws){\n apop_model * apop_varad_var(in, NULL);\n Apop_stopif(!in, return NAN, 0, \"NULL input model. Returning NaN.\");\n int apop_varad_var(draws, 1000);\nAPOP_VAR_ENDHEAD\n static int setup=0; if (!(setup++)){\n apop_entropy_vtable_add(norment, apop_normal);\n }\n apop_entropy_type e_fn = apop_entropy_vtable_get(in);\n if (e_fn) return e_fn(in);\n\n apop_data *d = apop_model_draws(in, draws);\n apop_data *lls = apop_map(d, .fn_rp=get_ll, .param=in);\n\n long double out = -apop_vector_mean(lls->vector);\n apop_data_free(d);\n apop_data_free(lls);\n return out;\n}\n\n/** Kullback-Leibler divergence.\n\nThis measure of the divergence of one distribution from another has the form \\f$ D(p,q)\n= \\sum_i \\ln(p_i/q_i) p_i \\f$. Notice that it is not a distance, because there is an\nasymmetry between \\f$p\\f$ and \\f$q\\f$, so one can expect that \\f$D(p, q) \\neq D(q, p)\\f$.\n\n \\param from the \\f$p\\f$ in the above formula. (No default; must not be \\c NULL)\n \\param to the \\f$q\\f$ in the above formula. (No default; must not be \\c NULL)\n \\param draw_ct If I do the calculation via random draws, how many? (Default = 1e5)\n \\param rng A \\c gsl_rng. If \\c NULL or number of threads is greater than 1, I'll take care of the RNG; see \\ref apop_rng_get_thread. (Default = \\c NULL)\n\nThis function can take empirical histogram-type models (\\ref apop_pmf) or continuous\nmodels like \\ref apop_loess or \\ref apop_normal.\n\nIf the \\c from distribution is a PMF (determined by checking whether its \\c p function\nis that of \\ref apop_pmf), then I'll step through it for the points in the summation.\n\n\\li If you have two empirical distributions in the form of \\ref apop_pmf, they must\nbe synced: if \\f$p_i>0\\f$ but \\f$q_i=0\\f$, then the function returns \\c GSL_NEGINF. If\napop_opts.verbose >=1 I print a message as well.\n\nIf the \\c from distribution is not a PMF, then I will take \\c draw_ct random draws\nfrom \\c from and evaluate at those points.\n\n\\li Set apop_opts.verbose = 3 for observation-by-observation info.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD long double apop_kl_divergence(apop_model *from, apop_model *to, int draw_ct, gsl_rng *rng){\n apop_model * apop_varad_var(from, NULL);\n apop_model * apop_varad_var(to, NULL);\n Apop_stopif(!from, return NAN, 0, \"The first model is NULL; returning NaN.\");\n Apop_stopif(!to, return NAN, 0, \"The second model is NULL.\");\n double apop_varad_var(draw_ct, 1e5);\n gsl_rng * apop_varad_var(rng, NULL);\nAPOP_VAR_ENDHEAD\n double div = 0;\n Apop_notify(3, \"p(from)\\tp(to)\\tfrom*log(from/to)\\n\");\n if (from->p == apop_pmf->p){\n apop_data *p = from->data;\n apop_pmf_settings *settings = Apop_settings_get_group(from, apop_pmf);\n Get_vmsizes(p); //maxsize\n OMP_for_reduce (+:div, int i=0; i < maxsize; i++){\n double pi = p->weights ? gsl_vector_get(p->weights, i)/settings->total_weight : 1./maxsize;\n if (!pi){\n Apop_notify(3, \"0\\t--\\t0\");\n continue;\n } //else:\n double qi = apop_p(Apop_r(p, i), to);\n Apop_notify(3,\"%g\\t%g\\t%g\", pi, qi, pi ? pi * log(pi/qi):0);\n Apop_stopif(!qi, div+=GSL_NEGINF; break, 1, \"The PMFs aren't synced: from-distribution has a value where \"\n \"to-distribution doesn't (which produces infinite divergence).\");\n div += pi * log(pi/qi);\n }\n } else { //the version with the RNG.\n Apop_stopif(!from->dsize, return GSL_NAN, 0, \"I need to make random draws from the 'from' model, \"\n \"but its dsize (draw size)==0. Returning NaN.\");\n OMP_for_reduce(+:div, int i=0; i < draw_ct; i++){\n double draw[from->dsize];\n apop_draw(draw, rng, from);\n gsl_matrix_view dm = gsl_matrix_view_array(draw, 1, from->dsize);\n double pi = apop_p(&(apop_data){.matrix=&(dm.matrix)}, from);\n double qi = apop_p(&(apop_data){.matrix=&(dm.matrix)}, to);\n double val = pi ? log(pi/qi): 0; //each row already has probability p_i\n Apop_notify(3,\"%g\\t%g\\t%g\", pi, qi, val);\n div += val;\n Apop_stopif(!qi, break, 1, \"From-distribution has a value where \"\n \"to-distribution doesn't (which produces infinite divergence).\");\n }\n div /= draw_ct; //div is an expected value of ln(pi/qi)\n }\n return div;\n}\n\n/** The multivariate generalization of the Gamma distribution.\n\\f[\n\\Gamma_p(a)=\n\\pi^{p(p-1)/4}\\prod_{j=1}^p\n\\Gamma\\left[ a+(1-j)/2\\right]. \\f]\n\nBecause \\f$\\Gamma(x)\\f$ is undefined for \\f$x\\in\\{0, -1, -2, ...\\}\\f$, this function returns \\c NAN when \\f$a+(1-j)/2\\f$ takes on one of those values.\n\nSee also \\ref apop_multivariate_lngamma, which is more numerically stable in most cases.\n*/\nlong double apop_multivariate_gamma(double a, int p){\n Apop_stopif(-(a+(1-p)/2) == (int)-(a+(1-p)/2) && a+(1-p)/2 <=0, return NAN, 1, \"Undefined when a + (1-p)/2 = 0, -1, -2, ... [you sent a=%g, p=%i]\", a, p);\n long double out = pow(M_PI, p*(p-1.)/4.);\n long double factor = 1;\n for (int i=1; i<=p; i++)\n factor *= gsl_sf_gamma(a+(1-i)/2.);\n return out * factor;\n}\n\n/** The log of the multivariate generalization of the Gamma; see also\n \\ref apop_multivariate_gamma.\n*/\nlong double apop_multivariate_lngamma(double a, int p){\n Apop_stopif(-(a+(1-p)/2) == (int)-(a+(1-p)/2) && a+(1-p)/2 <=0, return NAN, 1, \"Undefined when a + (1-p)/2 = 0, -1, -2, ... [you sent a=%g, p=%i]\", a, p);\n long double out = M_LNPI * p*(p-1.)/4.;\n for (int i=1; i<=p; i++)\n out += gsl_sf_lngamma(a+(1-i)/2.);\n return out;\n}\n\nstatic void find_eigens(gsl_matrix **subject, gsl_vector *eigenvals, gsl_matrix *eigenvecs){\n gsl_eigen_symmv_workspace * w = gsl_eigen_symmv_alloc((*subject)->size1);\n gsl_eigen_symmv(*subject, eigenvals, eigenvecs, w);\n gsl_eigen_symmv_free (w);\n gsl_matrix_free(*subject); *subject = NULL;\n}\n\nstatic void diagonal_copy(gsl_vector *v, gsl_matrix *m, char in_or_out){\n gsl_vector_view dv = gsl_matrix_diagonal(m);\n if (in_or_out == 'i') gsl_vector_memcpy(&(dv.vector), v);\n else gsl_vector_memcpy(v, &(dv.vector));\n}\n\nstatic double diagonal_size(gsl_matrix *m){\n gsl_vector_view dv = gsl_matrix_diagonal(m);\n return apop_sum(&dv.vector);\n}\n\nstatic double biggest_elmt(gsl_matrix *d){ \n return GSL_MAX(fabs(gsl_matrix_max(d)), fabs(gsl_matrix_min(d)));\n}\n\n/** Test whether the input matrix is positive semidefinite (PSD).\n\nA covariance matrix will always be PSD, so this function can tell you whether your matrix is a valid covariance matrix.\n\nConsider the 1x1 matrix in the upper left of the input, then the 2x2 matrix in the\nupper left, on up to the full matrix. If the matrix is PSD, then each of these has\na positive determinant. This function thus calculates \\f$N\\f$ determinants for an\n\\f$N\\f$x\\f$N\\f$ matrix.\n\n\\param m The matrix to test. If \\c NULL, I will return zero---not PSD.\n\\param semi If anything but \\c 's', check for positive definite, not semidefinite. (default 's')\n\nSee also \\ref apop_matrix_to_positive_semidefinite, which will change the input to something PSD.\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD int apop_matrix_is_positive_semidefinite(gsl_matrix *m, char semi){\n gsl_matrix * apop_varad_var(m, NULL);\n Apop_stopif(!m, return 0, 1, \"You gave me a NULL matrix. I will take this as not positive semidefinite; returning zero.\");\n char apop_varad_var(semi, 's');\nAPOP_VAR_ENDHEAD\n for (int i=1; i<= m->size1; i++){\n gsl_matrix mv =gsl_matrix_submatrix (m, 0, 0, i, i).matrix;\n double det = apop_matrix_determinant(&mv);\n if ((semi == 'd' && det <0) || det <=0)\n return 0;\n }\n return 1;\n}\n\nvoid vfabs(double *x){*x = fabs(*x);}\n\n/** This function takes in a matrix and converts it in place to the `closest' positive semidefinite matrix.\n\n\\param m On input, any matrix; on output, a positive semidefinite matrix. If \\c NULL, return \\c NaN and print an error.\n\\return the distance between the original and new matrices.\n\n\\li See also the test function \\ref apop_matrix_is_positive_semidefinite.\n\\li This function can be used as the core of a model constraint.\n\\li Adapted from the R Matrix package's nearPD, which is \nCopyright (2007) Jens Oehlschlägel [under the GPL].\n*/\ndouble apop_matrix_to_positive_semidefinite(gsl_matrix *m){\n Apop_stopif(!m, return NAN, 0, \"Got a NULL matrix. Returning NaN.\");\n if (apop_matrix_is_positive_semidefinite(m)) return 0; \n double diffsize=0, dsize;\n apop_data *qdq; \n gsl_matrix *d = apop_matrix_copy(m);\n gsl_matrix *original = apop_matrix_copy(m);\n double orig_diag_size = fabs(diagonal_size(d));\n int size = d->size1;\n gsl_vector *diag = gsl_vector_alloc(size);\n diagonal_copy(diag, d, 'o');\n apop_vector_apply(diag, vfabs);\n double origsize = biggest_elmt(d);\n do {\n //get eigenvals\n apop_data *eigenvecs = apop_data_alloc(size, size);\n gsl_vector *eigenvals = gsl_vector_calloc(size);\n gsl_matrix *junk_copy = apop_matrix_copy(d);\n find_eigens(&junk_copy, eigenvals, eigenvecs->matrix);//junk freed here.\n \n //prune positive only\n int j=0;\n int plussize = eigenvecs->matrix->size1;\n int *mask = calloc(eigenvals->size , sizeof(int));\n for (int i=0; i< eigenvals->size; i++)\n plussize -= \n mask[i] = (gsl_vector_get(eigenvals, i) <= 0);\n \n //construct Q = pruned eigenvals\n apop_data_rm_columns(eigenvecs, mask);\n if (!eigenvecs->matrix) break;\n \n //construct D = positive eigen diagonal\n apop_data *eigendiag = apop_data_calloc(0, plussize, plussize);\n for (int i=0; i< eigenvals->size; i++)\n if (!mask[i]) {\n apop_data_set(eigendiag, j, j, eigenvals->data[i]);\n j++;\n }\n\n // Our candidate is QDQ', symmetrized, with the old diagonal subbed in.\n apop_data *qd = apop_dot(eigenvecs, eigendiag);\n qdq = apop_dot(qd, eigenvecs, .form2='t');\n for (int i=0; i< qdq->matrix->size1; i++)\n for (int j=i+1; j< qdq->matrix->size1; j++){\n double avg = (apop_data_get(qdq, i, j) +apop_data_get(qdq, j, i)) /2.;\n apop_data_set(qdq, i, j, avg);\n apop_data_set(qdq, j, i, avg);\n }\n diagonal_copy(diag, qdq->matrix, 'i');\n \n // Evaluate progress, clean up.\n dsize = biggest_elmt(d);\n gsl_matrix_sub(d, qdq->matrix);\n diffsize = biggest_elmt(d);\n apop_data_free(qd); gsl_matrix_free(d);\n apop_data_free(eigendiag); free(mask);\n apop_data_free(eigenvecs); gsl_vector_free(eigenvals);\n d = qdq->matrix;\n qdq->matrix=NULL; apop_data_free(qdq); qdq = NULL;\n } while (diffsize/dsize > 1e-3);\n\n apop_data *eigenvecs = apop_data_alloc(size, size);\n gsl_vector *eigenvals = gsl_vector_calloc(size);\n find_eigens(&d, eigenvals, eigenvecs->matrix);//free d here.\n //make eigenvalues more positive\n double score =0;\n for (int i=0; i< eigenvals->size; i++){\n double v = gsl_vector_get(eigenvals, i);\n if (v < 1e-1){\n gsl_vector_set(eigenvals, i, 1e-1);\n score += 1e-1 - v;\n }\n }\n for (int i=0; i< size; i++)\n assert(eigenvals->data[i] >=0);\n //if (score){\n apop_data *eigendiag = apop_data_calloc(0, size, size);\n diagonal_copy(eigenvals, eigendiag->matrix, 'i');\n double new_diag_size = diagonal_size(eigendiag->matrix);\n gsl_matrix_scale(eigendiag->matrix, orig_diag_size/new_diag_size);\n apop_data *qd = apop_dot(eigenvecs, eigendiag);\n qdq = apop_dot(qd, eigenvecs, .form2='t');\n gsl_matrix_memcpy(m, qdq->matrix);\n apop_data_free(qd);\n apop_data_free(eigendiag);\n //}\n assert(apop_matrix_is_positive_semidefinite(m));\n apop_data_free(qdq); gsl_vector_free(diag);\n apop_data_free(eigenvecs); gsl_vector_free(eigenvals);\n gsl_matrix_sub(original, m);\n return biggest_elmt(original)/origsize;\n}\n" }, { "path": "apop_tests.m4.c", "content": "/** \\file apop_tests.c\t */\n/* Copyright (c) 2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. \n */\n#include \"apop_internal.h\"\n\nstatic apop_data * produce_t_test_output(int df, double stat, double diff){\n double pval, qval, two_tail;\n if (!gsl_isnan(stat)){\n pval = gsl_cdf_tdist_P(stat, df);\n qval = 1-pval;\n two_tail= 2*GSL_MIN(fabs(pval-.5),fabs(qval-0.5));\n } else {\n pval = GSL_NAN;\n qval = GSL_NAN;\n two_tail= GSL_NAN;\n }\n apop_data *out = apop_data_alloc();\n apop_data_add_named_elmt(out, \"mean left - right\", diff);\n apop_data_add_named_elmt(out, \"t statistic\", stat);\n apop_data_add_named_elmt(out, \"df\", df);\n apop_data_add_named_elmt(out, \"p value, 1 tail\", GSL_MIN(pval,qval));\n apop_data_add_named_elmt(out, \"confidence, 1 tail\", 1 - GSL_MIN(pval,qval));\n apop_data_add_named_elmt(out, \"p value, 2 tail\", 1- two_tail);\n apop_data_add_named_elmt(out, \"confidence, 2 tail\", two_tail);\n return out;\n}\n\n/** Answers the question: with what confidence can I say that the means of these two columns of data are different?\n\nIf \\c apop_opts.verbose is >=1, then display some information to stdout, like the mean/var/count for both vectors and the t statistic.\n\n\\param a one column of data\n\\param b another column of data\n\\return an \\ref apop_data set with the following elements:
\n mean left - right: the difference in means; if positive, first vector has larger mean, and one-tailed test is testing \\f$L > R\\f$, else reverse if negative.
\n t statistic: used for the test
\n df: degrees of freedom
\n p value, 1 tail: the p-value for a one-tailed test that one vector mean is greater than the other.
\n confidence, 1 tail: 1- p value.
\n p value, 2 tail: the p-value for the two-tailed test that left mean = right mean.
\n confidence, 2 tail: 1-p value\n\nExample usage:\n\\code\ngsl_vector *L = apop_query_to_vector(\"select * from data where sex='M'\");\ngsl_vector *R = apop_query_to_vector(\"select * from data where sex='F'\");\napop_data *test_out = apop_t_test(L, R);\nprintf(\"Reject the null hypothesis of no difference between M and F with %g%% confidence\\n\", apop_data_get(test_out, .rowname=\"confidence, 2 tail\"));\n\\endcode\n\n\\see \\ref apop_paired_t_test, which answers the question: with what confidence can I\nsay that the mean difference between the two columns is zero?\n*/\napop_data *\tapop_t_test(gsl_vector *a, gsl_vector *b){\n int a_count = a->size,\n b_count = b->size;\n double a_avg = apop_vector_mean(a);\n double a_var = (a_count > 1) ? apop_vector_var(a) : 0,\n b_avg = apop_vector_mean(b),\n b_var = b_count > 1 ? apop_vector_var(b): 0,\n stat = (a_avg - b_avg)/ sqrt(\n (b_count > 1 ? b_var/(b_count-1) : 0) \n + (a_count > 1 ? a_var/(a_count-1) : 0) \n );\n if (apop_opts.verbose >=1){\n printf(\"1st avg: %g; 1st std dev: %g; 1st count: %i.\\n\", a_avg, sqrt(a_var), a_count);\n printf(\"2st avg: %g; 2st std dev: %g; 2nd count: %i.\\n\", b_avg, sqrt(b_var), b_count);\n printf(\"t-statistic: %g.\\n\", stat);\n }\n int df = a_count+b_count-2;\n return produce_t_test_output(df, stat, a_avg - b_avg);\n}\n\n/** Answers the question: with what confidence can I say that the mean difference between the two columns is zero?\n\nIf apop_opts.verbose >=2, then display some information, like the mean/var/count for both vectors and the t statistic, to stderr.\n\n\\param a A column of data\n\\param b A matched column of data\n\\return an \\ref apop_data set with the following elements:\n mean left - right: the difference in means; if positive, first vector has larger mean, and one-tailed test is testing \\f$L > R\\f$, else reverse if negative.
\n t statistic: used for the test
\n df: degrees of freedom
\n p value, 1 tail: the p-value for a one-tailed test that one vector mean is greater than the other.
\n confidence, 1 tail: 1- p value.
\n p value, 2 tail: the p-value for the two-tailed test that left mean = right mean.
\n confidence, 2 tail: 1-p value\n\n\\see \\ref apop_t_test for an example, and for when the element-by-element difference between the vectors has no sensible interpretation.\n*/\napop_data * apop_paired_t_test(gsl_vector *a, gsl_vector *b){\n gsl_vector *diff = gsl_vector_alloc(a->size);\n gsl_vector_memcpy(diff, a);\n gsl_vector_sub(diff, b);\n int count = a->size; \n double avg = apop_vector_mean(diff),\n var = apop_vector_var(diff),\n stat = avg/ sqrt(var/(count-1));\n gsl_vector_free(diff);\n Apop_notify(2, \"avg diff: %g; diff std dev: %g; count: %i; t-statistic: %g.\\n\", avg, sqrt(var), count, stat);\n return produce_t_test_output(count-1, stat, avg);\n}\n\n/** Runs an F-test specified by \\c q and \\c c. See\n the chapter on hypothesis testing in Modeling With Data, p 309, which will tell you that:\n \\f[{N-K\\over q}\n {({\\bf Q}'\\hat\\beta - {\\bf c})' [{\\bf Q}' ({\\bf X}'{\\bf X})^{-1} {\\bf Q}]^{-1} ({\\bf Q}' \\hat\\beta - {\\bf c})\n \\over {\\bf u}' {\\bf u} } \\sim F_{q,N-K},\\f]\n and that's what this function is based on.\n\n\\param est An \\ref apop_model that you have already calculated. (No default)\n\\param contrast An \\ref apop_data set whose matrix represents \\f${\\bf Q}\\f$ and whose\n vector represents \\f${\\bf c}\\f$. Each row represents a hypothesis. (Defaults:\n if matrix is \\c NULL, it is set to the identity matrix with the top row missing. If\n the vector is \\c NULL, it is set to a zero matrix of length equal to the height of\n the contrast matrix. Thus, if the entire \\c apop_data set is NULL or omitted, we are\n testing the hypothesis that all but \\f$\\beta_1\\f$ are zero.)\n\n\\return An \\c apop_data set with a few variants on the confidence with which we can reject the joint hypothesis.\n\\todo There should be a way to get OLS and GLS to store \\f$(X'X)^{-1}\\f$. In fact, if you did GLS, this is invalid, because you need \\f$(X'\\Sigma X)^{-1}\\f$, and I didn't ask for \\f$\\Sigma\\f$.\n\n\\exception out->error='a' Allocation error.\n\\exception out->error='d' dimension-matching error.\n\\exception out->error='i' matrix inversion error.\n\\exception out->error='m' GSL math error.\n\n\\li There are two approaches to an \\f$F\\f$-test: the ANOVA approach, which is typically\n built around the claim that all effects but the mean are zero; and the more general\n regression form, which allows for any set of linear claims about the data. If you send\n a \\c NULL contrast set, I will generate the set of linear contrasts that are equivalent\n to the ANOVA-type approach. This is why the top row of the default \\f${\\bf Q}\\f$\n matrix is missing: there is no hypothesis test about the coefficient for the\n constant term. See the example below.\n\\li This function uses the \\ref designated syntax for inputs.\n\n\\include f_test.c\n*/\nAPOP_VAR_HEAD apop_data * apop_f_test (apop_model *est, apop_data *contrast){\n apop_model *apop_varad_var(est, NULL)\n Nullcheck_m(est, NULL);\n Nullcheck_d(est->data, NULL);\n apop_data * apop_varad_var(contrast, NULL);\n int free_data=0, free_matrix=0, free_vector=0;\n if (!contrast) contrast = apop_data_alloc(), free_data=1;\n if (!contrast->matrix) {\n int size = est->parameters->vector->size;\n contrast->matrix= gsl_matrix_calloc(size - 1, size);\n for (int i=1; i< size; i++)\n apop_data_set(contrast, i-1, i, 1);\n }\n if (!contrast->vector) contrast->vector = gsl_vector_calloc(contrast->matrix->size1), free_vector=1;\n\n apop_data *out = apop_f_test_base(est, contrast);\n if (free_data) {apop_data_free(contrast); return out;}\n if (free_matrix) gsl_matrix_free(contrast->matrix);\n if (free_vector) gsl_vector_free(contrast->vector);\n return out;\nAPOP_VAR_ENDHEAD\n apop_data *out = apop_data_alloc();\n Asprintf(&out->names->title, \"F test\");\n size_t contrast_ct = contrast->vector->size;\n Apop_stopif(contrast->matrix->size1 != contrast_ct, out->error='d'; return out,\n 0, \"I counted %zu contrasts by the size of either contrast->vector or \"\n \"est->parameters->vector->size, but you gave me a matrix with %zu rows. Those should match.\"\n , contrast_ct, contrast->matrix->size1);\n double f_stat, pval;\n\n Get_vmsizes(est->data); //msize1, msize2\n int data_df = msize1 - contrast_ct;\n\n //Find (\\underbar x)'(\\underbar x), where (\\underbar x) = the data with means removed\n long double means[msize2];\n for (int i=1; i< msize2; i++)\n means[i] = apop_vector_mean(Apop_cv(est->data, i));\n means[0]=0;// don't screw with the ones column.\n apop_data *xpx = apop_data_alloc(msize2, msize2);\n Apop_stopif(xpx->error, apop_data_free(xpx); out->error='a'; return out, 0, \"allocation error\");\n for (int i=0; i< msize2; i++)\n for (int j=0; j< msize2; j++){ //at this loop, we calculate one cell in the dot prouct\n long double total = 0;\n for (int c=0; cdata->matrix, c, i) - means[i])\n *(gsl_matrix_get(est->data->matrix, c, j) - means[j]);\n apop_data_set(xpx, i, j, total);\n }\n\n apop_data xpxinv = (apop_data){.matrix=apop_matrix_inverse(xpx->matrix)};\n Apop_stopif(!xpxinv.matrix, out->error='i'; return out, 0, \"inversion of X'X error\");\n apop_data *qprimexpxinv = apop_dot(contrast, &xpxinv, 'm', 'm');\n apop_data *qprimexpxinvq = apop_dot(qprimexpxinv, contrast, 'm', 't');\n Apop_stopif(qprimexpxinvq->error || qprimexpxinv->error, out->error='m'; return out, 0, \"broken dot\");\n apop_data qprimexpxinvqinv = (apop_data){.matrix=apop_matrix_inverse(qprimexpxinvq->matrix)};\n Apop_stopif(!qprimexpxinvqinv.matrix, out->error='i'; return out, 0, \"inversion of Q'(X'X)^{-1}Q error\");\n apop_data_free(qprimexpxinvq);\n apop_data_free(qprimexpxinv);\n apop_data *qprimebeta = apop_dot(contrast, est->parameters, 'm', 'v');\n Apop_stopif(qprimebeta->error, out->error='m'; return out, 0, \"broken dot\");\n gsl_vector_sub(qprimebeta->vector, contrast->vector);\n apop_data *qprimebetaminusc_qprimexpxinvqinv = apop_dot(&qprimexpxinvqinv, qprimebeta, .form2='v');\n Apop_stopif(qprimebetaminusc_qprimexpxinvqinv->error, out->error='m'; return out, 0, \"broken dot\");\n gsl_blas_ddot(qprimebeta->vector, qprimebetaminusc_qprimexpxinvqinv->vector, &f_stat);\n apop_data_free(xpx);\n apop_data_free(qprimebeta);\n apop_data_free(qprimebetaminusc_qprimexpxinvqinv);\n\n apop_data *r_sq_list = apop_estimate_coefficient_of_determination (est);\n double variance = apop_data_get(r_sq_list, .rowname=\"sse\");\n f_stat *= data_df / (variance * contrast_ct);\n pval = (contrast_ct > 0 && data_df > 0) ? gsl_cdf_fdist_Q(f_stat, contrast_ct, data_df): GSL_NAN; \n\n apop_data_add_named_elmt(out, \"F statistic\", f_stat);\n apop_data_add_named_elmt(out, \"p value\", pval);\n apop_data_add_named_elmt(out, \"confidence\", 1- pval);\n apop_data_add_named_elmt(out, \"df1\", contrast_ct);\n apop_data_add_named_elmt(out, \"df2\", data_df);\n return out;\n}\n\nstatic double one_chi_sq(apop_data *d, int row, int col, int n){\n double rowexp = apop_vector_sum(Apop_rv(d, row))/n;\n double colexp = apop_vector_sum(Apop_cv(d, col))/n;\n double observed = apop_data_get(d, row, col);\n double expected = n * rowexp * colexp;\n return gsl_pow_2(observed - expected)/expected; \n}\n\n/** Run a Chi-squared test on an ANOVA table, i.e., an NxN table with the null hypothesis that all cells are equally likely.\n\n \\param d The input data, which is a crosstab of various elements. They don't have to sum to one.\n\\return A \\ref apop_data set including elements named\n \"chi squared statistic\", \"df\", and \"p value\". Retrieve via,\n e.g., apop_data_get(out, .rowname=\"p value\").\n \\see apop_test_fisher_exact\n*/\napop_data * apop_test_anova_independence(apop_data *d){\n Apop_stopif(!d || !d->matrix, return NULL, 0, \"You sent me data with no matrix element. Returning NULL.\");\n double total = 0;\n //You can have a one-column or one-row matrix if you want; else df = (rows-1)*(cols-1)\n double df = d->matrix->size1==1 ? d->matrix->size2-1 : d->matrix->size2 == 1 ? d->matrix->size1 \n : (d->matrix->size1 - 1)* (d->matrix->size2 - 1);\n Apop_stopif(!df, return NULL, 0, \"You sent a degenerate matrix. Returning NULL.\");\n int n = apop_matrix_sum(d->matrix);\n for (size_t row=0; row matrix->size1; row++)\n for (size_t col=0; col matrix->size2; col++)\n total += one_chi_sq(d, row, col, n);\n apop_data *out = apop_data_alloc();\n double chisq = gsl_cdf_chisq_Q(total, df);\n apop_data_add_named_elmt(out, \"chi squared statistic\", total);\n apop_data_add_named_elmt(out, \"df\", df);\n apop_data_add_named_elmt(out, \"p value\", chisq);\n return out;\n}\n\n\nstatic apop_data* apop_anova_one_way(char *table, char *data, char *grouping){\n //ANOVA has always just been a process of filling in a form, and\n //that's what this function does.\n apop_data *out = apop_data_calloc(3, 6);\n apop_name_add(out->names, \"sum of squares\", 'c');\n apop_name_add(out->names, \"df\", 'c');\n apop_name_add(out->names, \"mean squares\", 'c');\n apop_name_add(out->names, \"F ratio\", 'c');\n apop_name_add(out->names, \"p value\", 'c');\n apop_name_add(out->names, \"confidence\", 'c');\n apop_name_add(out->names, grouping, 'r');\n apop_name_add(out->names, \"residual\", 'r');\n apop_name_add(out->names, \"total\", 'r');\n \n //total sum of squares:\n apop_data* tss = apop_query_to_data(\"select var_pop(%s), count(*) from %s\", data, table);\n Apop_stopif(!tss, apop_return_data_error('q'), 0, \"Query 'select var_pop(%s), count(*) from %s' returned NULL. Does that look right to you?\", data, table);\n apop_data_set(out, 2, 0, apop_data_get(tss, 0, 0)*apop_data_get(tss, 0, 1)); //total sum of squares\n double total_df = apop_data_get(tss, 0, 1);\n apop_data_set(out, 2, 1, apop_data_get(tss, 0, 1)); //total df.\n\n //within group sum of squares:\n apop_data* wss = apop_query_to_data(\"select var_pop(%s), count(*) from %s group by %s\", data, table, grouping);\n double group_df = wss->matrix->size1-1;\n apop_data_set(out, 0, 0, apop_data_get(wss, 0, 0)*group_df); //within sum of squares\n apop_data_set(out, 0, 1, group_df);\n\n //residuals are just total-wss\n apop_data_set(out, 1, 0, apop_data_get(out, 2, 0) - apop_data_get(out, 0,0)); //residual sum of squares\n double residual_df = total_df - group_df;\n apop_data_set(out, 1, 1, residual_df); //residual df\n\n apop_data_set(out, 0, 2, apop_data_get(out, 0, 0)/apop_data_get(out, 0, 1));//mean SS within\n apop_data_set(out, 1, 2, apop_data_get(out, 1, 0)/apop_data_get(out, 1, 1));//mean SS residual\n\n apop_data_set(out, 0, 3, apop_data_get(out, 0, 2)/apop_data_get(out, 1, 2));//F ratio\n apop_data_set(out, 0, 4, gsl_cdf_fdist_P(apop_data_get(out, 0, 3), group_df, residual_df));//pval\n apop_data_set(out, 0, 5, 1- apop_data_get(out, 0, 4));//confidence\n\n apop_data_free(tss);\n apop_data_free(wss);\n return out;\n}\n\n/** This function produces a traditional one- or two-way ANOVA table. It\n works from data in an SQL table, using queries of a form like select\n data from table group by grouping1, grouping2.\n\n \\param table The table to be queried. Anything that can go in an SQL from clause is OK, so this can be a plain table name or a temp table specification like (select ... ), with parens.\n \\param data The name of the column holding the count or other such data\n \\param grouping1 The name of the first column by which to group data\n \\param grouping2 If this is \\c NULL, then the function will return a one-way ANOVA. Otherwise, the name of the second column by which to group data in a two-way ANOVA.\n */\nAPOP_VAR_HEAD apop_data* apop_anova(char *table, char *data, char *grouping1, char *grouping2){\n char *apop_varad_var(table, NULL)\n Apop_stopif(!table, return NULL, 0, \"I need the name of a table in the SQL database.\");\n if (!strchr(table, <|')'|>)) //if you found ()s, then it is a temp table spec.\n Apop_stopif(!apop_table_exists(table), return NULL, 0, \"I couldn't find the table %s in the database.\", table);\n char *apop_varad_var(data, NULL)\n Apop_stopif(!data, return NULL, 0, \"I need the name of the column in the %s table with the count or other data.\", table);\n char *apop_varad_var(grouping1, NULL)\n Apop_stopif(!data, return NULL, 0, \"I need at least grouping1, a column in the %s table.\", table);\n char *apop_varad_var(grouping2, NULL)\nAPOP_VAR_ENDHEAD\n apop_data *first = apop_anova_one_way(table, data, grouping1);\n Apop_stopif(first->error, return first, 0, \"Error (%c) running one-way ANOVA.\", first->error);\n if (!grouping2) return first;\n apop_data *second = apop_anova_one_way(table, data, grouping2);\n char *joined = NULL;\n Asprintf(&joined, \"%s, %s\", grouping1, grouping2);\n apop_data *interaction = apop_anova_one_way(table, data, joined);\n apop_data *out = apop_data_calloc(5, 6);\n apop_name_stack(out->names, first->names, 'c');\n apop_data_add_names(out, 'r', first->names->row[0], second->names->row[0],\n \"interaction\", \"residual\", \"total\");\n gsl_vector *firstrow = Apop_rv(first, 0);\n gsl_vector *secondrow = Apop_rv(second, 0);\n gsl_vector *interrow = Apop_rv(interaction, 0);\n gsl_matrix_set_row(out->matrix, 0, firstrow);\n gsl_matrix_set_row(out->matrix, 1, secondrow);\n gsl_matrix_set_row(out->matrix, 2, interrow);\n gsl_matrix_set_row(out->matrix, 4, Apop_rv(first, 2));\n \n //residuals are just total-wss\n apop_data_set(out, 3, 0, apop_data_get(out, 4, 0) \n - gsl_vector_get(firstrow, 0) - gsl_vector_get(secondrow, 0) - gsl_vector_get(interrow, 0)); //residual sum of squares\n double residual_df = apop_data_get(out, 4, 1) \n - gsl_vector_get(firstrow, 1) - gsl_vector_get(secondrow, 1) - gsl_vector_get(interrow, 1); //residual df\n apop_data_set(out, 3, 1, residual_df);\n\n apop_data_set(out, 3, 2, apop_data_get(out, 3, 0)/apop_data_get(out, 3, 1));//mean SS residual\n\n apop_data_set(out, 0, 3, apop_data_get(out, 0, 2)/apop_data_get(out, 3, 2));//F ratio\n apop_data_set(out, 0, 4, gsl_cdf_fdist_P(apop_data_get(out, 0, 3), gsl_vector_get(firstrow, 1), residual_df));//pval\n apop_data_set(out, 0, 5, 1- apop_data_get(out, 0, 4));//confidence\n\n apop_data_set(out, 1, 3, apop_data_get(out, 1, 2)/apop_data_get(out, 3, 2));//F ratio\n apop_data_set(out, 1, 4, gsl_cdf_fdist_P(apop_data_get(out, 1, 3), gsl_vector_get(secondrow, 1), residual_df));//pval\n apop_data_set(out, 1, 5, 1- apop_data_get(out, 1, 4));//confidence\n\n apop_data_set(out, 2, 3, apop_data_get(out, 2, 2)/apop_data_get(out, 3, 2));//F ratio\n apop_data_set(out, 2, 4, gsl_cdf_fdist_P(apop_data_get(out, 2, 3), gsl_vector_get(interrow, 1), residual_df));//pval\n apop_data_set(out, 2, 5, 1- apop_data_get(out, 2, 4));//confidence\n\n free(joined);\n apop_data_free(first);\n apop_data_free(second);\n apop_data_free(interaction);\n return out;\n}\n\n/** This is a convenience function to do the lookup of a given statistic along a given\ndistribution. You give me a statistic, its (hypothesized) distribution, and whether\nto use the upper tail, lower tail, or both. I will return the odds of a Type I error\ngiven the model---in statistician jargon, the \\f$p\\f$-value. [Type I error: odds of\nrejecting the null hypothesis when it is true.]\n\n For example, \n \\code\n apop_test(1.3);\n \\endcode\n\nwill return the density of the standard Normal distribution that is more than 1.3 from zero. \nIf this function returns a small value, we can be confident that the statistic is significant. Or, \n \\code\n apop_test(1.3, \"t\", 10, .tail='u');\n \\endcode\n\nwill give the appropriate odds for an upper-tailed test using the \\f$t\\f$-distribution with 10 degrees of freedom (e.g., a \\f$t\\f$-test of the null hypothesis that the statistic is less than or equal to zero).\n\nSeveral more distributions are supported; see below.\n\n \\li For a two-tailed test (the default), this returns the density outside the range. I'll only do this for symmetric distributions.\n \\li For an upper-tail test ('u'), this returns the density above the cutoff\n \\li For a lower-tail test ('l'), this returns the density below the cutoff \n \n\\param statistic The scalar value to be tested. \n\\param distribution The name of the distribution; see below.\n\\param p1 The first parameter for the distribution; see below.\n\\param p2 The second parameter for the distribution; see below.\n\\param tail 'u' = upper tail; 'l' = lower tail; anything else = two-tailed. (default = two-tailed)\n\n\\return The odds of a Type I error given the model (the \\f$p\\f$-value).\n\nHere are the distributions you can use and their parameters.\n\n\\c \"normal\" or \\c \"gaussian\" \n\\li p1=\\f$\\mu\\f$, p2=\\f$\\sigma\\f$\n\\li default (0, 1)\n\n\\c \"lognormal\" \n\\li p1=\\f$\\mu\\f$, p2=\\f$\\sigma\\f$\n\\li default (0, 1) \n\\li Remember, \\f$\\mu\\f$ and \\f$\\sigma\\f$ refer to the Normal one would get after exponentiation\n\\li One-tailed tests only\n\n\\c \"uniform\" \n\\li p1=lower edge, p2=upper edge\n\\li default (0, 1)\n\\li two-tailed tests are run relative to the center, (p1+p2)/2.\n\n\\c \"t\" \n\\li p1=df\n\\li no default\n\n\\c \"chi squared\", \\c \"chi\", \\c \"chisq\": \n\\li p1=df\n\\li no default\n\\li One-tailed tests only; default='u' (\\f$p\\f$-value for typical cases)\n\n\\c \"f\" \n\\li p1=df1, p2=df2\n\\li no default\n\\li One-tailed tests only\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD double apop_test(double statistic, char *distribution, double p1, double p2, char tail){\n double apop_varad_var(statistic, 0);\n char* apop_varad_var(distribution, NULL);\n double apop_varad_var(p1, 0);\n double apop_varad_var(p2, 0);\n int is_chi = !strcasecmp(distribution, \"chi squared\")|| !strcasecmp(distribution, \"chi\")\n || !strcasecmp(distribution, \"chisq\");\n Apop_stopif(!strcasecmp(distribution, \"f\") && (!p1 || !p2), return NAN, 0, \"I need both a p1 and p2 parameter specifying the degrees of freedom.\");\n Apop_stopif((!strcasecmp(distribution, \"t\") || !strcasecmp(distribution, \"f\") || is_chi)\n && !p1, return NAN, 0, \"I need a p1 parameter specifying the degrees of freedom.\");\n if (!p2 && (!distribution || !strcasecmp(distribution, \"normal\") || !strcasecmp(distribution, \"gaussian\") ))\n p2 = 1;\n if (!p2 && p1 >= 0 && !strcasecmp(distribution, \"uniform\"))\n p2 = 1;\n\n char apop_varad_var(tail, 0);\n if (!tail) tail = is_chi ? 'u' : 'a';\nAPOP_VAR_ENDHEAD\n //This is a long and boring function. I am aware that there are\n //clever way to make it shorter.\n if (!distribution || !strcasecmp(distribution, \"normal\") || !strcasecmp(distribution, \"gaussian\") ){\n if (tail == 'u')\n return gsl_cdf_gaussian_Q(p1-statistic, p2);\n else if (tail == 'l')\n return gsl_cdf_gaussian_P(p1-statistic, p2);\n else\n return 2 * gsl_cdf_gaussian_Q(fabs(p1-statistic), p2);\n }\n else if (!strcasecmp(distribution, \"lognormal\")){\n if (tail == 'u')\n return gsl_cdf_lognormal_Q(statistic, p1, p2);\n else if (tail == 'l')\n return gsl_cdf_lognormal_P(statistic, p1, p2);\n else\n Apop_assert(0, \"A two-tailed test doesn't really make sense for the lognormal. Please specify either tail= 'u' or tail= 'l'.\");\n }\n else if (!strcasecmp(distribution, \"t\")){\n if (tail == 'u')\n return gsl_cdf_tdist_Q(statistic, p1);\n else if (tail == 'l')\n return gsl_cdf_tdist_P(statistic, p1);\n else\n return 2 * gsl_cdf_tdist_Q(fabs(statistic), p1);\n }\n else if (!strcasecmp(distribution, \"f\")){\n if (tail == 'u')\n return gsl_cdf_fdist_Q(statistic, p1, p2);\n else if (tail == 'l')\n return gsl_cdf_fdist_P(statistic, p1, p2);\n else\n Apop_assert(0, \"A two-tailed test doesn't really make sense for the %s. Please specify either tail= 'u' or tail= 'l'.\", distribution);\n }\n else if (!strcasecmp(distribution, \"chi squared\")|| !strcasecmp(distribution, \"chi\")\n || !strcasecmp(distribution, \"chisq\")){\n if (tail == 'u')\n return gsl_cdf_chisq_Q(statistic, p1);\n else if (tail == 'l')\n return gsl_cdf_chisq_P(statistic, p1);\n else\n Apop_assert(0, \"A two-tailed test doesn't really make sense for the %s. Please specify either tail= 'u' or tail= 'l'.\", distribution);\n }\n else if (!strcasecmp(distribution, \"uniform\")){\n if (tail == 'u')\n return gsl_cdf_flat_Q(statistic, p1, p2);\n else if (tail == 'l')\n return gsl_cdf_flat_P(statistic, p1, p2);\n else\n return 2 * gsl_cdf_flat_Q(fabs(statistic - (p1+p2)/2.), p1, p2);\n }\n Apop_assert(0, \"Sorry, but I don't recognize %s as a distribution\", distribution);\n}\n" }, { "path": "apop_update.m4.c", "content": "/** \\file \n The \\ref apop_update function. */ \n/* Copyright (c) 2006--2009, 2014 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\n/* This file in four parts:\n --an apop_model named product, purpose-built for apop_update to send to apop_model_metropolis\n --apop_mcmc settings and their defaults.\n --apop_update and its equipment, which has three cases:\n --conjugates, in which case see the functions\n --call Metropolis\n*/\n\n\n/* This will be used by apop_update to send to apop_mcmc below.\n\n To set it up, add a more pointer to an array of two models, the prior and likelihood. \n The total likelihood of a data point is (likelihood these parameters are drawn from\n prior)*(likelihood of these parameters and the data set using the likelihood fn)\n*/\nstatic long double product_ll(apop_data *d, apop_model *m){\n apop_model **pl = m->more;\n gsl_vector *v = apop_data_pack(m->parameters);\n apop_data_unpack(v, pl[1]->parameters);\n gsl_vector_free(v);\n return apop_log_likelihood(m->parameters, pl[0]) + apop_log_likelihood(d, pl[1]);\n}\n\nstatic long double product_constraint(apop_data *data, apop_model *m){\n apop_model **pl = m->more;\n gsl_vector *v = apop_data_pack(m->parameters);\n apop_data_unpack(v, pl[1]->parameters);\n gsl_vector_free(v);\n return pl[1]->constraint(data, pl[1]);\n}\n\napop_model *product = &(apop_model){\"product of two models\", \n .log_likelihood=product_ll, .constraint=product_constraint};\n\n\n///////////the conjugate table\n\nstatic apop_model *betabinom(apop_data *data, apop_model *prior, apop_model *likelihood){\n apop_model *outp = apop_model_copy(prior);\n if (!data && likelihood->parameters){\n double n = likelihood->parameters->vector->data[0];\n double p = likelihood->parameters->vector->data[1];\n *gsl_vector_ptr(outp->parameters->vector, 0) += n*p;\n *gsl_vector_ptr(outp->parameters->vector, 1) += n*(1-p);\n } else {\n gsl_vector *hits = Apop_cv(data, 1);\n gsl_vector *misses = Apop_cv(data, 0);\n *gsl_vector_ptr(outp->parameters->vector, 0) += apop_sum(hits);\n *gsl_vector_ptr(outp->parameters->vector, 1) += apop_sum(misses);\n }\n return outp;\n}\n\ndouble countup(double in){return in!=0;}\n\nstatic apop_model *betabernie(apop_data *data, apop_model *prior, apop_model *likelihood){\n apop_model *outp = apop_model_copy(prior);\n Get_vmsizes(data);//tsize\n double sum = apop_map_sum(data, .fn_d=countup, .part='a');\n *gsl_vector_ptr(outp->parameters->vector, 0) += sum;\n *gsl_vector_ptr(outp->parameters->vector, 1) += tsize - sum;\n return outp;\n}\n\nstatic apop_model *gammaexpo(apop_data *data, apop_model *prior, apop_model *likelihood){\n apop_model *outp = apop_model_copy(prior);\n Get_vmsizes(data); //maxsize\n *gsl_vector_ptr(outp->parameters->vector, 0) += maxsize;\n apop_data_set(outp->parameters, 1, .val=1./\n (1./apop_data_get(outp->parameters, 1) \n + (data->matrix ? apop_matrix_sum(data->matrix) : 0)\n + (data->vector ? apop_sum(data->vector) : 0)));\n return outp;\n}\n\nstatic apop_model *gammapoisson(apop_data *data, apop_model *prior, apop_model *likelihood){\n /* Posterior alpha = alpha_0 + sum x; posterior beta = beta_0/(beta_0*n + 1) */\n apop_model *outp = apop_model_copy(prior);\n Get_vmsizes(data); //vsize, msize1,maxsize\n *gsl_vector_ptr(outp->parameters->vector, 0) +=\n (vsize ? apop_sum(data->vector): 0) +\n (msize1 ? apop_matrix_sum(data->matrix): 0);\n\n double *beta = gsl_vector_ptr(outp->parameters->vector, 1);\n *beta = *beta/(*beta * maxsize + 1);\n return outp;\n}\n\nstatic apop_model *normnorm(apop_data *data, apop_model *prior, apop_model *likelihood){\n/*\noutput \\f$(\\mu, \\sigma) = (\\frac{\\mu_0}{\\sigma_0^2} + \\frac{\\sum_{i=1}^n x_i}{\\sigma^2})/(\\frac{1}{\\sigma_0^2} + \\frac{n}{\\sigma^2}), (\\frac{1}{\\sigma_0^2} + \\frac{n}{\\sigma^2})^{-1}\\f$\n\nThat is, the output is weighted by the number of data points for the\nlikelihood. If you give me a parametrized normal, with no data, then I'll take the weight to be \\f$n=1\\f$. \n*/\n double mu_like, var_like;\n long int n;\n apop_model *outp = apop_model_copy(prior);\n apop_prep(data, outp);\n long double mu_pri = prior->parameters->vector->data[0];\n long double var_pri = gsl_pow_2(prior->parameters->vector->data[1]);\n if (!data && likelihood->parameters){\n mu_like = likelihood->parameters->vector->data[0];\n var_like = gsl_pow_2(likelihood->parameters->vector->data[1]);\n n = 1;\n } else {\n n = data->matrix->size1 * data->matrix->size2;\n apop_matrix_mean_and_var(data->matrix, &mu_like, &var_like);\n }\n gsl_vector_set(outp->parameters->vector, 0, (mu_pri/var_pri + n*mu_like/var_like)/(1/var_pri + n/var_like));\n gsl_vector_set(outp->parameters->vector, 1, pow((1/var_pri + n/var_like), -.5));\n return outp;\n}\n\n/** Take in a prior and likelihood distribution, and output a posterior distribution.\n\n\\li This function first checks a table of conjugate distributions for the pair you sent\nin. If the models are listed on the table, then the function returns a corresponding\nclosed-form model with updated parameters.\n\n\\li If the parameters aren't in the table of conjugate, and the prior distribution has\na \\c p or \\c log_likelihood element, then use \\ref apop_model_metropolis to generate\nthe posterior. If you expect MCMC to run, you may add an \\ref apop_mcmc_settings\ngroup to your prior to control the details of the search. See also the \\ref\napop_model_metropolis documentation.\n\n\\li If the prior does not have a \\c p or \\c log_likelihood but does have a \\c draw\nelement, then make draws from the prior and weight them by the \\c p given by the\nlikelihood distribution. This is not a rejection sampling method, so the burnin\nis ignored.\n\n\\param data The input data, that will be used by the likelihood function (default = \\c NULL.)\n\\param prior The prior \\ref apop_model. If the system needs to\nestimate the posterior via MCMC, this needs to have a \\c log_likelihood or \\c p method. (No default, must not be \\c NULL.)\n\\param likelihood The likelihood \\ref apop_model. If the system needs to\nestimate the posterior via MCMC, this needs to have a \\c log_likelihood or \\c p method (ll preferred). (No default, must not be \\c NULL.)\n\\param rng A \\c gsl_rng, already initialized (e.g., via \\ref apop_rng_alloc). (default: an RNG from \\ref apop_rng_get_thread)\n\\return an \\ref apop_model struct representing the posterior, with updated parameters. \n\n\n\\li In all cases, the output is a \\ref apop_model that can be used as the input to this\nfunction, so you can chain Bayesian updating procedures.\n\\li Here are the conjugate distributions currently defined:\n\n\n\n Likelihood Notes \n\n \\ref apop_binomial \"Binomial\" \n\n \\ref apop_bernoulli \"Bernoulli\" \n\n \\ref apop_gamma \"Gamma\" Gamma likelihood represents the distribution of \\f$\\lambda^{-1}\\f$, not plain \\f$\\lambda\\f$\n\n \\ref apop_normal \"Normal\" Assumes prior with fixed \\f$\\sigma\\f$; updates distribution for \\f$\\mu\\f$\n\n \\ref apop_poisson \"Poisson\" Uses sum and size of the data \n\n
Prior
\\ref apop_beta \"Beta\"
\\ref apop_beta \"Beta\"
\\ref apop_exponential \"Exponential\"
\\ref apop_normal \"Normal\"
\\ref apop_gamma \"Gamma\"
\n\nHere is a test function that compares the output via conjugate table and via\nMetropolis-Hastings sampling: \n\\include test_updating.c\n\n\\li The conjugate table is stored using a vtable; see \\ref vtables for details. If you\nare writing a new vtable entry, the typedef new functions must conform to and the hash\nused for lookups are:\n\n\\code\ntypedef apop_model *(*apop_update_type)(apop_data *, apop_model , apop_model);\n#define apop_update_hash(m1, m2) ((size_t)(m1).draw + (size_t)((m2).log_likelihood ? (m2).log_likelihood : (m2).p)*33)\n\\endcode\n\n\\li This function uses the \\ref designated syntax for inputs.\n*/\nAPOP_VAR_HEAD apop_model * apop_update(apop_data *data, apop_model *prior, apop_model *likelihood, gsl_rng *rng){\n apop_data *apop_varad_var(data, NULL);\n apop_model *apop_varad_var(prior, NULL);\n apop_model *apop_varad_var(likelihood, NULL);\n gsl_rng *apop_varad_var(rng, apop_rng_get_thread(-1));\nAPOP_VAR_END_HEAD\n static int setup=0; if (!(setup++)){\n apop_update_vtable_add(betabinom, apop_beta, apop_binomial);\n apop_update_vtable_add(betabernie, apop_beta, apop_bernoulli);\n apop_update_vtable_add(gammaexpo, apop_gamma, apop_exponential);\n apop_update_vtable_add(gammapoisson, apop_gamma, apop_poisson);\n apop_update_vtable_add(normnorm, apop_normal, apop_normal);\n }\n apop_update_type conj = apop_update_vtable_get(prior, likelihood);\n if (conj) return conj(data, prior, likelihood);\n\n apop_mcmc_settings *s = apop_settings_get_group(prior, apop_mcmc);\n\n apop_prep(NULL, prior); //probably a no-op\n apop_prep(data, likelihood); //probably a no-op\n gsl_vector *pack = apop_data_pack(likelihood->parameters);\n int tsize = pack->size;\n gsl_vector_free(pack);\n Apop_stopif(prior->dsize != tsize, \n return apop_model_copy(&(apop_model){.error='d'}),\n 0, \"Size of a draw from the prior does not match \"\n \"the size of the likelihood's parameters (%i != %i).%s\",\n prior->dsize, tsize, \n (tsize > prior->dsize) ? \n \" Perhaps use apop_model_fix_params to reduce the \"\n \"likelihood's parameter count?\" : \"\");\n if (prior->p || prior->log_likelihood){\n apop_model *p = apop_model_copy(product);\n //pending revision, a memory leak:\n p->more = malloc(sizeof(apop_model*)*2);\n ((apop_model**)p->more)[0] = apop_model_copy(prior);\n ((apop_model**)p->more)[1] = apop_model_copy(likelihood);\n p->more_size = sizeof(apop_model*) * 2;\n p->parameters = apop_data_alloc(prior->dsize);\n p->data = data;\n if (s) apop_settings_copy_group(p, prior, \"apop_mcmc\");\n apop_model *out = apop_model_metropolis(data, rng, p); \n return out;\n }\n\n Apop_stopif(!prior->draw, return NULL, 0, \"prior does not have a .p, .log_likelihood, or .draw element. I am stumped. Returning NULL.\");\n\n if (!s) s = Apop_model_add_group(prior, apop_mcmc);\n\n gsl_vector *draw = gsl_vector_alloc(tsize);\n apop_data *out = apop_data_alloc(s->periods, tsize);\n out->weights = gsl_vector_alloc(s->periods);\n\n apop_draw(draw->data, rng, prior); //set starting point.\n apop_data_unpack(draw, likelihood->parameters);\n\n for (int i=0; i< s->periods; i++){\n newdraw:\n apop_draw(draw->data, rng, prior);\n apop_data_unpack(draw, likelihood->parameters);\n long double p = apop_p(data, likelihood);\n\n Apop_notify(3, \"p=%Lg for parameters:\\t\", p);\n if (apop_opts.verbose >=3) apop_data_print(likelihood->parameters);\n\n Apop_stopif(gsl_isnan(p), goto newdraw,\n 1, \"Trouble evaluating the \"\n \"likelihood function at vector beginning with %g. \"\n \"Throwing it out and trying again.\\n\"\n , likelihood->parameters->vector->data[0]);\n apop_data_pack(likelihood->parameters, Apop_rv(out, i));\n gsl_vector_set(out->weights, i, p);\n }\n apop_model *outp = apop_estimate(out, apop_pmf);\n gsl_vector_free(draw);\n return outp;\n}\n" }, { "path": "apop_vtables.c", "content": "#include \n#include \n#include \"apop_internal.h\" //just for OMP_critical\n\n#ifdef _OPENMP\n#include \n#define lock omp_set_lock(&v->mutex);\n#define unlock omp_unset_lock(&v->mutex);\n#else\n#define lock \n#define unlock \n#endif\n\n/** \\cond doxy_ignore */\ntypedef struct {\n size_t hash;\n void *fn;\n} apop_vtable_elmt_s;\n\ntypedef struct {\n char const *name;\n unsigned long int hashed_name;\n int elmt_ct;\n apop_vtable_elmt_s *elmts;\n#ifdef _OPENMP\n omp_lock_t mutex;\n#endif\n} apop_vtable_s;\n\napop_vtable_s *vtable_list;\nint ignore_me;\n/** \\endcond */ //End of Doxygen ignore.\n\n//The Dan J Bernstein string hashing algorithm.\nstatic unsigned long apop_settings_hash(char const *str){\n unsigned long int hash = 5381;\n char c;\n while ((c = *str++)) hash = hash*33 + c;\n return hash;\n}\n\nstatic apop_vtable_s *find_tab(unsigned long h, int *ctr){\n apop_vtable_s *v = vtable_list;\n *ctr = 0;\n for ( ; v->hashed_name; (*ctr)++, v++) if (v->hashed_name== h) break;\n return v;\n}\n\n//return 0 = found; removed\n//return 1 = not found; no-op\nint apop_vtable_drop(char const *tabname, unsigned long hash){\n if (!vtable_list) return 1;\n unsigned long h = apop_settings_hash(tabname);\n apop_vtable_s *v = find_tab(h, &ignore_me);\n\n lock\n for (int i=0; i< v->elmt_ct; i++)\n if (hash == v->elmts[i].hash) {\n memmove(v->elmts+i, v->elmts+i+1, sizeof(apop_vtable_elmt_s)*(v->elmt_ct-i));\n v->elmt_ct--;\n unlock\n return 0;\n }\n unlock\n return 1;\n}\n\nint apop_vtable_add(char const *tabname, void *fn_in, unsigned long hash){\n if (!vtable_list){vtable_list = calloc(1, sizeof(apop_vtable_s));}\n\n unsigned long h = apop_settings_hash(tabname);\n int ctr;\n apop_vtable_s *v;\n\n\n //add a table if need be.\n OMP_critical (new_vtable)\n {\n v = find_tab(h, &ctr);\n\n if (!v->hashed_name){\n vtable_list = realloc(vtable_list, (ctr+2)* sizeof(apop_vtable_s));\n vtable_list[ctr] = (apop_vtable_s){.elmts=calloc(1, sizeof(apop_vtable_elmt_s))};\n vtable_list[ctr+1] = (apop_vtable_s){ };\n #ifdef _OPENMP\n omp_init_lock(&vtable_list[ctr].mutex);\n omp_set_lock(&vtable_list[ctr].mutex);\n #endif\n vtable_list[ctr].name = tabname;\n vtable_list[ctr].hashed_name = h;\n v = vtable_list+ctr;\n unlock\n }\n }\n\n lock\n //If this hash is already present, don't re-add. \n for (int i=0; i< v->elmt_ct; i++) if (hash == v->elmts[i].hash) {unlock; return 0;}\n\n //insert\n v->elmts = realloc(v->elmts, (++(v->elmt_ct))* sizeof(apop_vtable_elmt_s));\n v->elmts[v->elmt_ct-1] = (apop_vtable_elmt_s){.hash=hash, .fn=fn_in};\n unlock\n return 0;\n}\n\nvoid *apop_vtable_get(char const *tabname, unsigned long hash){\n if (!vtable_list) return NULL;\n unsigned long thash = apop_settings_hash(tabname);\n apop_vtable_s *v = find_tab(thash, &ignore_me);\n if (!v->hashed_name) return NULL;\n\n lock\n for (int i=0; i< v->elmt_ct; i++)\n if (hash == v->elmts[i].hash) {unlock; return v->elmts[i].fn;}\n unlock\n return NULL;\n}\n" }, { "path": "asprintf.c", "content": " /** \\cond doxy_ignore */\n/* Formatted output to strings.\n Copyright (C) 1999, 2002 Free Software Foundation, Inc.\n\n This program is free software; you can redistribute it and/or modify\n it under the terms of the GNU General Public License as published by\n the Free Software Foundation; either version 2, or (at your option)\n any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU General Public License for more details.\n\n You should have received a copy of the GNU General Public License along\n with this program; if not, write to the Free Software Foundation,\n Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. */\n\n/* Mashed into one file by BK */\n\n/* Tell glibc's to provide a prototype for snprintf().\n This must come before because may include\n , and once has been included, it's too late. */\n#ifndef _GNU_SOURCE\n# define _GNU_SOURCE 1\n#endif\n\n/* vsprintf with automatic memory allocation. */\n\n#include \n#include \n#include \n\n#ifdef HAVE_CONFIG_H\n# include \n#endif\n\n#if HAVE_ASPRINTF\n#else\n\n#ifndef __attribute__\n/* This feature is available in gcc versions 2.5 and later. */\n# if __GNUC__ < 2 || (__GNUC__ == 2 && __GNUC_MINOR__ < 5) || __STRICT_ANSI__\n# define __attribute__(Spec) /* empty */\n# endif\n/* The __-protected variants of `format' and `printf' attributes\n are accepted by gcc versions 2.6.4 (effectively 2.7) and later. */\n# if __GNUC__ < 2 || (__GNUC__ == 2 && __GNUC_MINOR__ < 7)\n# define __format__ format\n# define __printf__ printf\n# endif\n#endif\n\n#ifdef\t__cplusplus\nextern \"C\" {\n#endif\n\n/* Write formatted output to a string dynamically allocated with malloc().\n If the memory allocation succeeds, store the address of the string in\n *RESULT and return the number of resulting bytes, excluding the trailing\n NUL. Upon memory allocation error, or some other error, return -1. */\nextern int asprintf (char **result, const char *format, ...)\n __attribute__ ((__format__ (__printf__, 2, 3)));\nextern int vasprintf (char **result, const char *format, va_list args)\n __attribute__ ((__format__ (__printf__, 2, 0)));\n\n//from vasnprintf\nextern char * asnprintf (char *resultbuf, size_t *lengthp, const char *format, ...)\n __attribute__ ((__format__ (__printf__, 3, 4)));\nextern char * vasnprintf (char *resultbuf, size_t *lengthp, const char *format, va_list args)\n __attribute__ ((__format__ (__printf__, 3, 0)));\n\n#ifdef\t__cplusplus\n}\n#endif\n\n\n/* Decomposed printf argument list. */\n\n/* Get wint_t. */\n#ifdef HAVE_WINT_T\n# include \n#endif\n\n/* Argument types */\ntypedef enum {\n TYPE_NONE,\n TYPE_SCHAR,\n TYPE_UCHAR,\n TYPE_SHORT,\n TYPE_USHORT,\n TYPE_INT,\n TYPE_UINT,\n TYPE_LONGINT,\n TYPE_ULONGINT,\n#ifdef HAVE_LONG_LONG\n TYPE_LONGLONGINT,\n TYPE_ULONGLONGINT,\n#endif\n TYPE_DOUBLE,\n#ifdef HAVE_LONG_DOUBLE\n TYPE_LONGDOUBLE,\n#endif\n TYPE_CHAR,\n#ifdef HAVE_WINT_T\n TYPE_WIDE_CHAR,\n#endif\n TYPE_STRING,\n#ifdef HAVE_WCHAR_T\n TYPE_WIDE_STRING,\n#endif\n TYPE_POINTER,\n TYPE_COUNT_SCHAR_POINTER,\n TYPE_COUNT_SHORT_POINTER,\n TYPE_COUNT_INT_POINTER,\n TYPE_COUNT_LONGINT_POINTER\n#ifdef HAVE_LONG_LONG\n, TYPE_COUNT_LONGLONGINT_POINTER\n#endif\n} arg_type;\n\n/* Polymorphic argument */\ntypedef struct {\n arg_type type;\n union\n {\n signed char\t\t\ta_schar;\n unsigned char\t\ta_uchar;\n short\t\t\ta_short;\n unsigned short\t\ta_ushort;\n int\t\t\t\ta_int;\n unsigned int\t\ta_uint;\n long int\t\t\ta_longint;\n unsigned long int\t\ta_ulongint;\n#ifdef HAVE_LONG_LONG\n long long int\t\ta_longlongint;\n unsigned long long int\ta_ulonglongint;\n#endif\n float\t\t\ta_float;\n double\t\t\ta_double;\n#ifdef HAVE_LONG_DOUBLE\n long double\t\t\ta_longdouble;\n#endif\n int\t\t\t\ta_char;\n#ifdef HAVE_WINT_T\n wint_t\t\t\ta_wide_char;\n#endif\n const char*\t\t\ta_string;\n#ifdef HAVE_WCHAR_T\n const wchar_t*\t\ta_wide_string;\n#endif\n void*\t\t\ta_pointer;\n signed char *\t\ta_count_schar_pointer;\n short *\t\t\ta_count_short_pointer;\n int *\t\t\ta_count_int_pointer;\n long int *\t\t\ta_count_longint_pointer;\n#ifdef HAVE_LONG_LONG\n long long int *\t\ta_count_longlongint_pointer;\n#endif\n }\n a;\n}\nargument;\n\ntypedef struct {\n size_t count;\n argument *arg;\n}\narguments;\n\n/* Fetch the arguments, putting them into a. */\n#ifdef STATIC\nSTATIC\n#else\nextern\n#endif\nint printf_fetchargs (va_list args, arguments *a);\n\n/* Parse printf format string. */\n\n/* Flags */\n#define FLAG_GROUP\t 1\t/* ' flag */\n#define FLAG_LEFT\t 2\t/* - flag */\n#define FLAG_SHOWSIGN\t 4\t/* + flag */\n#define FLAG_SPACE\t 8\t/* space flag */\n#define FLAG_ALT\t16\t/* # flag */\n#define FLAG_ZERO\t32\n\n/* arg_index value indicating that no argument is consumed. */\n#define ARG_NONE\t(~(size_t)0)\n\n/* A parsed directive. */\ntypedef struct {\n const char* dir_start;\n const char* dir_end;\n int flags;\n const char* width_start;\n const char* width_end;\n size_t width_arg_index;\n const char* precision_start;\n const char* precision_end;\n size_t precision_arg_index;\n char conversion; /* d i o u x X f e E g G c s p n U % but not C S */\n size_t arg_index;\n}\nchar_directive;\n\n/* A parsed format string. */\ntypedef struct {\n size_t count;\n char_directive *dir;\n size_t max_width_length;\n size_t max_precision_length;\n}\nchar_directives;\n\n\n/* Parses the format string. Fills in the number N of directives, and fills\n in directives[0], ..., directives[N-1], and sets directives[N].dir_start\n to the end of the format string. Also fills in the arg_type fields of the\n arguments and the needed count of arguments. */\n#ifdef STATIC\nSTATIC\n#else\nextern\n#endif\nint printf_parse (const char *format, char_directives *d, arguments *a);\n\n/*end headers */\n\nchar * asnprintf (char *resultbuf, size_t *lengthp, const char *format, ...) {\n va_list args;\n char *result;\n\n va_start (args, format);\n result = vasnprintf (resultbuf, lengthp, format, args);\n va_end (args);\n return result;\n}\n\nint asprintf (char **resultp, const char *format, ...) {\n va_list args;\n int result;\n\n va_start (args, format);\n result = vasprintf (resultp, format, args);\n va_end (args);\n return result;\n}\n\n#ifdef STATIC\nSTATIC\n#endif\nint printf_fetchargs (va_list args, arguments *a) {\n size_t i;\n argument *ap;\n\n for (i = 0, ap = &a->arg[0]; i < a->count; i++, ap++)\n switch (ap->type)\n {\n case TYPE_SCHAR:\n\tap->a.a_schar = va_arg (args, /*signed char*/ int);\n\tbreak;\n case TYPE_UCHAR:\n\tap->a.a_uchar = va_arg (args, /*unsigned char*/ int);\n\tbreak;\n case TYPE_SHORT:\n\tap->a.a_short = va_arg (args, /*short*/ int);\n\tbreak;\n case TYPE_USHORT:\n\tap->a.a_ushort = va_arg (args, /*unsigned short*/ int);\n\tbreak;\n case TYPE_INT:\n\tap->a.a_int = va_arg (args, int);\n\tbreak;\n case TYPE_UINT:\n\tap->a.a_uint = va_arg (args, unsigned int);\n\tbreak;\n case TYPE_LONGINT:\n\tap->a.a_longint = va_arg (args, long int);\n\tbreak;\n case TYPE_ULONGINT:\n\tap->a.a_ulongint = va_arg (args, unsigned long int);\n\tbreak;\n#ifdef HAVE_LONG_LONG\n case TYPE_LONGLONGINT:\n\tap->a.a_longlongint = va_arg (args, long long int);\n\tbreak;\n case TYPE_ULONGLONGINT:\n\tap->a.a_ulonglongint = va_arg (args, unsigned long long int);\n\tbreak;\n#endif\n case TYPE_DOUBLE:\n\tap->a.a_double = va_arg (args, double);\n\tbreak;\n#ifdef HAVE_LONG_DOUBLE\n case TYPE_LONGDOUBLE:\n\tap->a.a_longdouble = va_arg (args, long double);\n\tbreak;\n#endif\n case TYPE_CHAR:\n\tap->a.a_char = va_arg (args, int);\n\tbreak;\n#ifdef HAVE_WINT_T\n case TYPE_WIDE_CHAR:\n\tap->a.a_wide_char = va_arg (args, wint_t);\n\tbreak;\n#endif\n case TYPE_STRING:\n\tap->a.a_string = va_arg (args, const char *);\n\tbreak;\n#ifdef HAVE_WCHAR_T\n case TYPE_WIDE_STRING:\n\tap->a.a_wide_string = va_arg (args, const wchar_t *);\n\tbreak;\n#endif\n case TYPE_POINTER:\n\tap->a.a_pointer = va_arg (args, void *);\n\tbreak;\n case TYPE_COUNT_SCHAR_POINTER:\n\tap->a.a_count_schar_pointer = va_arg (args, signed char *);\n\tbreak;\n case TYPE_COUNT_SHORT_POINTER:\n\tap->a.a_count_short_pointer = va_arg (args, short *);\n\tbreak;\n case TYPE_COUNT_INT_POINTER:\n\tap->a.a_count_int_pointer = va_arg (args, int *);\n\tbreak;\n case TYPE_COUNT_LONGINT_POINTER:\n\tap->a.a_count_longint_pointer = va_arg (args, long int *);\n\tbreak;\n#ifdef HAVE_LONG_LONG\n case TYPE_COUNT_LONGLONGINT_POINTER:\n\tap->a.a_count_longlongint_pointer = va_arg (args, long long int *);\n\tbreak;\n#endif\n default:\n\t/* Unknown type. */\n\treturn -1;\n }\n return 0;\n}\n\n/* Get intmax_t. */\n#if HAVE_STDINT_H_WITH_UINTMAX\n# include \n#endif\n#if HAVE_INTTYPES_H_WITH_UINTMAX\n# include \n#endif\n\n/* malloc(), realloc(), free(). */\n#include \n\n/* xsize.h -- Checked size_t computations. */\n\n#ifndef _XSIZE_H\n#define _XSIZE_H\n\n/* Get SIZE_MAX. */\n#include \n#if HAVE_STDINT_H\n# include \n#endif\n\n/* The size of memory objects is often computed through expressions of\n type size_t. Example:\n void* p = malloc (header_size + n * element_size).\n These computations can lead to overflow. When this happens, malloc()\n returns a piece of memory that is way too small, and the program then\n crashes while attempting to fill the memory.\n To avoid this, the functions and macros in this file check for overflow.\n The convention is that SIZE_MAX represents overflow.\n malloc (SIZE_MAX) is not guaranteed to fail -- think of a malloc\n implementation that uses mmap --, it's recommended to use size_overflow_p()\n or size_in_bounds_p() before invoking malloc().\n The example thus becomes:\n size_t size = xsum (header_size, xtimes (n, element_size));\n void *p = (size_in_bounds_p (size) ? malloc (size) : NULL);\n*/\n\n/* Convert an arbitrary value >= 0 to type size_t. */\n#define xcast_size_t(N) \\\n ((N) <= SIZE_MAX ? (size_t) (N) : SIZE_MAX)\n\n/* Sum of two sizes, with overflow check. */\nstatic inline size_t\n#if __GNUC__ >= 3\n__attribute__ ((__pure__))\n#endif\nxsum (size_t size1, size_t size2) {\n size_t sum = size1 + size2;\n return (sum >= size1 ? sum : SIZE_MAX);\n}\n\n/* Sum of three sizes, with overflow check. */\nstatic inline size_t\n#if __GNUC__ >= 3\n__attribute__ ((__pure__))\n#endif\nxsum3 (size_t size1, size_t size2, size_t size3) {\n return xsum (xsum (size1, size2), size3);\n}\n\n/* Sum of four sizes, with overflow check. */\nstatic inline size_t\n#if __GNUC__ >= 3\n__attribute__ ((__pure__))\n#endif\nxsum4 (size_t size1, size_t size2, size_t size3, size_t size4) {\n return xsum (xsum (xsum (size1, size2), size3), size4);\n}\n\n/* Maximum of two sizes, with overflow check. */\nstatic inline size_t\n#if __GNUC__ >= 3\n__attribute__ ((__pure__))\n#endif\nxmax (size_t size1, size_t size2) {\n /* No explicit check is needed here, because for any n:\n max (SIZE_MAX, n) == SIZE_MAX and max (n, SIZE_MAX) == SIZE_MAX. */\n return (size1 >= size2 ? size1 : size2);\n}\n\n/* Multiplication of a count with an element size, with overflow check.\n The count must be >= 0 and the element size must be > 0.\n This is a macro, not an inline function, so that it works correctly even\n when N is of a wider tupe and N > SIZE_MAX. */\n#define xtimes(N, ELSIZE) \\\n ((N) <= SIZE_MAX / (ELSIZE) ? (size_t) (N) * (ELSIZE) : SIZE_MAX)\n\n/* Check for overflow. */\n#define size_overflow_p(SIZE) \\\n ((SIZE) == SIZE_MAX)\n/* Check against overflow. */\n#define size_in_bounds_p(SIZE) \\\n ((SIZE) != SIZE_MAX)\n\n#endif /* _XSIZE_H */\n\n\n#if WIDE_CHAR_VERSION\n# define PRINTF_PARSE wprintf_parse\n# define CHAR_T wchar_t\n# define DIRECTIVE wchar_t_directive\n# define DIRECTIVES wchar_t_directives\n#else\n# define PRINTF_PARSE printf_parse\n# define CHAR_T char\n# define DIRECTIVE char_directive\n# define DIRECTIVES char_directives\n#endif\n\n#ifdef STATIC\nSTATIC\n#endif\nint\nPRINTF_PARSE (const CHAR_T *format, DIRECTIVES *d, arguments *a) {\n const CHAR_T *cp = format;\t\t/* pointer into format */\n size_t arg_posn = 0;\t\t/* number of regular arguments consumed */\n size_t d_allocated;\t\t\t/* allocated elements of d->dir */\n size_t a_allocated;\t\t\t/* allocated elements of a->arg */\n size_t max_width_length = 0;\n size_t max_precision_length = 0;\n\n d->count = 0;\n d_allocated = 1;\n d->dir = malloc (d_allocated * sizeof (DIRECTIVE));\n if (d->dir == NULL)\n /* Out of memory. */\n return -1;\n\n a->count = 0;\n a_allocated = 0;\n a->arg = NULL;\n\n#define REGISTER_ARG(_index_,_type_) \\\n {\t\t\t\t\t\t\t\t\t\\\n size_t n = (_index_);\t\t\t\t\t\t\\\n if (n >= a_allocated)\t\t\t\t\t\t\\\n {\t\t\t\t\t\t\t\t\t\\\n\tsize_t memory_size;\t\t\t\t\t\t\\\n\targument *memory;\t\t\t\t\t\t\\\n\t\t\t\t\t\t\t\t\t\\\n\ta_allocated = xtimes (a_allocated, 2);\t\t\t\t\\\n\tif (a_allocated <= n)\t\t\t\t\t\t\\\n\t a_allocated = xsum (n, 1);\t\t\t\t\t\\\n\tmemory_size = xtimes (a_allocated, sizeof (argument));\t\t\\\n\tif (size_overflow_p (memory_size))\t\t\t\t\\\n\t /* Overflow, would lead to out of memory. */\t\t\t\\\n\t goto error;\t\t\t\t\t\t\t\\\n\tmemory = (a->arg\t\t\t\t\t\t\\\n\t\t ? realloc (a->arg, memory_size)\t\t\t\\\n\t\t : malloc (memory_size));\t\t\t\t\\\n\tif (memory == NULL)\t\t\t\t\t\t\\\n\t /* Out of memory. */\t\t\t\t\t\t\\\n\t goto error;\t\t\t\t\t\t\t\\\n\ta->arg = memory;\t\t\t\t\t\t\\\n }\t\t\t\t\t\t\t\t\t\\\n while (a->count <= n)\t\t\t\t\t\t\\\n a->arg[a->count++].type = TYPE_NONE;\t\t\t\t\\\n if (a->arg[n].type == TYPE_NONE)\t\t\t\t\t\\\n a->arg[n].type = (_type_);\t\t\t\t\t\\\n else if (a->arg[n].type != (_type_))\t\t\t\t\\\n /* Ambiguous type for positional argument. */\t\t\t\\\n goto error;\t\t\t\t\t\t\t\\\n }\n\n while (*cp != '\\0')\n {\n CHAR_T c = *cp++;\n if (c == '%')\n\t{\n\t size_t arg_index = ARG_NONE;\n\t DIRECTIVE *dp = &d->dir[d->count];/* pointer to next directive */\n\n\t /* Initialize the next directive. */\n\t dp->dir_start = cp - 1;\n\t dp->flags = 0;\n\t dp->width_start = NULL;\n\t dp->width_end = NULL;\n\t dp->width_arg_index = ARG_NONE;\n\t dp->precision_start = NULL;\n\t dp->precision_end = NULL;\n\t dp->precision_arg_index = ARG_NONE;\n\t dp->arg_index = ARG_NONE;\n\n\t /* Test for positional argument. */\n\t if (*cp >= '0' && *cp <= '9') {\n\t const CHAR_T *np;\n\n\t for (np = cp; *np >= '0' && *np <= '9'; np++)\n ;\n\t if (*np == '$') {\n size_t n = 0;\n\n for (np = cp; *np >= '0' && *np <= '9'; np++)\n n = xsum (xtimes (n, 10), *np - '0');\n if (n == 0)\n /* Positional argument 0. */\n goto error;\n if (size_overflow_p (n))\n /* n too large, would lead to out of memory later. */\n goto error;\n arg_index = n - 1;\n cp = np + 1;\n }\n\t }\n\n\t /* Read the flags. */\n\t for (;;) {\n\t if (*cp == '\\'') {\n dp->flags |= FLAG_GROUP;\n cp++;\n\t\t } else if (*cp == '-') {\n dp->flags |= FLAG_LEFT;\n cp++;\n\t\t } else if (*cp == '+') {\n dp->flags |= FLAG_SHOWSIGN;\n cp++;\n\t\t } else if (*cp == ' ') {\n dp->flags |= FLAG_SPACE;\n cp++;\n\t\t } else if (*cp == '#') {\n dp->flags |= FLAG_ALT;\n cp++;\n\t\t } else if (*cp == '0') {\n dp->flags |= FLAG_ZERO;\n cp++;\n\t\t } else\n\t\t break;\n }\n\n\t /* Parse the field width. */\n\t if (*cp == '*')\n\t {\n\t dp->width_start = cp;\n\t cp++;\n\t dp->width_end = cp;\n\t if (max_width_length < 1)\n\t\tmax_width_length = 1;\n\n\t /* Test for positional argument. */\n\t if (*cp >= '0' && *cp <= '9')\n\t\t{\n\t\t const CHAR_T *np;\n\n\t\t for (np = cp; *np >= '0' && *np <= '9'; np++)\n\t\t ;\n\t\t if (*np == '$')\n\t\t {\n\t\t size_t n = 0;\n\n\t\t for (np = cp; *np >= '0' && *np <= '9'; np++)\n\t\t\tn = xsum (xtimes (n, 10), *np - '0');\n\t\t if (n == 0)\n\t\t\t/* Positional argument 0. */\n\t\t\tgoto error;\n\t\t if (size_overflow_p (n))\n\t\t\t/* n too large, would lead to out of memory later. */\n\t\t\tgoto error;\n\t\t dp->width_arg_index = n - 1;\n\t\t cp = np + 1;\n\t\t }\n\t\t}\n\t if (dp->width_arg_index == ARG_NONE)\n\t\t{\n\t\t dp->width_arg_index = arg_posn++;\n\t\t if (dp->width_arg_index == ARG_NONE)\n\t\t /* arg_posn wrapped around. */\n\t\t goto error;\n\t\t}\n\t REGISTER_ARG (dp->width_arg_index, TYPE_INT);\n\t }\n\t else if (*cp >= '0' && *cp <= '9')\n\t {\n\t size_t width_length;\n\n\t dp->width_start = cp;\n\t for (; *cp >= '0' && *cp <= '9'; cp++)\n\t\t;\n\t dp->width_end = cp;\n\t width_length = dp->width_end - dp->width_start;\n\t if (max_width_length < width_length)\n\t\tmax_width_length = width_length;\n\t }\n\n\t /* Parse the precision. */\n\t if (*cp == '.')\n\t {\n\t cp++;\n\t if (*cp == '*')\n\t\t{\n\t\t dp->precision_start = cp - 1;\n\t\t cp++;\n\t\t dp->precision_end = cp;\n\t\t if (max_precision_length < 2)\n\t\t max_precision_length = 2;\n\n\t\t /* Test for positional argument. */\n\t\t if (*cp >= '0' && *cp <= '9')\n\t\t {\n\t\t const CHAR_T *np;\n\n\t\t for (np = cp; *np >= '0' && *np <= '9'; np++)\n\t\t\t;\n\t\t if (*np == '$')\n\t\t\t{\n\t\t\t size_t n = 0;\n\n\t\t\t for (np = cp; *np >= '0' && *np <= '9'; np++)\n\t\t\t n = xsum (xtimes (n, 10), *np - '0');\n\t\t\t if (n == 0)\n\t\t\t /* Positional argument 0. */\n\t\t\t goto error;\n\t\t\t if (size_overflow_p (n))\n\t\t\t /* n too large, would lead to out of memory\n\t\t\t later. */\n\t\t\t goto error;\n\t\t\t dp->precision_arg_index = n - 1;\n\t\t\t cp = np + 1;\n\t\t\t}\n\t\t }\n\t\t if (dp->precision_arg_index == ARG_NONE)\n\t\t {\n\t\t dp->precision_arg_index = arg_posn++;\n\t\t if (dp->precision_arg_index == ARG_NONE)\n\t\t\t/* arg_posn wrapped around. */\n\t\t\tgoto error;\n\t\t }\n\t\t REGISTER_ARG (dp->precision_arg_index, TYPE_INT);\n\t\t}\n\t else\n\t\t{\n\t\t size_t precision_length;\n\n\t\t dp->precision_start = cp - 1;\n\t\t for (; *cp >= '0' && *cp <= '9'; cp++)\n\t\t ;\n\t\t dp->precision_end = cp;\n\t\t precision_length = dp->precision_end - dp->precision_start;\n\t\t if (max_precision_length < precision_length)\n\t\t max_precision_length = precision_length;\n\t\t}\n\t }\n\n\t {\n\t arg_type type;\n\n\t /* Parse argument type/size specifiers. */\n\t {\n\t int flags = 0;\n\n\t for (;;)\n\t\t{\n\t\t if (*cp == 'h')\n\t\t {\n\t\t flags |= (1 << (flags & 1));\n\t\t cp++;\n\t\t }\n\t\t else if (*cp == 'L')\n\t\t {\n\t\t flags |= 4;\n\t\t cp++;\n\t\t }\n\t\t else if (*cp == 'l')\n\t\t {\n\t\t flags += 8;\n\t\t cp++;\n\t\t }\n#ifdef HAVE_INTMAX_T\n\t\t else if (*cp == 'j')\n\t\t {\n\t\t if (sizeof (intmax_t) > sizeof (long))\n\t\t\t{\n\t\t\t /* intmax_t = long long */\n\t\t\t flags += 16;\n\t\t\t}\n\t\t else if (sizeof (intmax_t) > sizeof (int))\n\t\t\t{\n\t\t\t /* intmax_t = long */\n\t\t\t flags += 8;\n\t\t\t}\n\t\t cp++;\n\t\t }\n#endif\n\t\t else if (*cp == 'z' || *cp == 'Z')\n\t\t {\n\t\t /* 'z' is standardized in ISO C 99, but glibc uses 'Z'\n\t\t\t because the warning facility in gcc-2.95.2 understands\n\t\t\t only 'Z' (see gcc-2.95.2/gcc/c-common.c:1784). */\n\t\t if (sizeof (size_t) > sizeof (long))\n\t\t\t{\n\t\t\t /* size_t = long long */\n\t\t\t flags += 16;\n\t\t\t}\n\t\t else if (sizeof (size_t) > sizeof (int))\n\t\t\t{\n\t\t\t /* size_t = long */\n\t\t\t flags += 8;\n\t\t\t}\n\t\t cp++;\n\t\t }\n\t\t else if (*cp == 't')\n\t\t {\n\t\t if (sizeof (ptrdiff_t) > sizeof (long))\n\t\t\t{\n\t\t\t /* ptrdiff_t = long long */\n\t\t\t flags += 16;\n\t\t\t}\n\t\t else if (sizeof (ptrdiff_t) > sizeof (int))\n\t\t\t{\n\t\t\t /* ptrdiff_t = long */\n\t\t\t flags += 8;\n\t\t\t}\n\t\t cp++;\n\t\t }\n\t\t else\n\t\t break;\n\t\t}\n\n\t /* Read the conversion character. */\n\t c = *cp++;\n\t switch (c)\n\t\t{\n\t\tcase 'd': case 'i':\n#ifdef HAVE_LONG_LONG\n\t\t if (flags >= 16 || (flags & 4))\n\t\t type = TYPE_LONGLONGINT;\n\t\t else\n#endif\n\t\t if (flags >= 8)\n\t\t type = TYPE_LONGINT;\n\t\t else if (flags & 2)\n\t\t type = TYPE_SCHAR;\n\t\t else if (flags & 1)\n\t\t type = TYPE_SHORT;\n\t\t else\n\t\t type = TYPE_INT;\n\t\t break;\n\t\tcase 'o': case 'u': case 'x': case 'X':\n#ifdef HAVE_LONG_LONG\n\t\t if (flags >= 16 || (flags & 4))\n\t\t type = TYPE_ULONGLONGINT;\n\t\t else\n#endif\n\t\t if (flags >= 8)\n\t\t type = TYPE_ULONGINT;\n\t\t else if (flags & 2)\n\t\t type = TYPE_UCHAR;\n\t\t else if (flags & 1)\n\t\t type = TYPE_USHORT;\n\t\t else\n\t\t type = TYPE_UINT;\n\t\t break;\n\t\tcase 'f': case 'F': case 'e': case 'E': case 'g': case 'G':\n\t\tcase 'a': case 'A':\n#ifdef HAVE_LONG_DOUBLE\n\t\t if (flags >= 16 || (flags & 4))\n\t\t type = TYPE_LONGDOUBLE;\n\t\t else\n#endif\n\t\t type = TYPE_DOUBLE;\n\t\t break;\n\t\tcase 'c':\n\t\t if (flags >= 8)\n#ifdef HAVE_WINT_T\n\t\t type = TYPE_WIDE_CHAR;\n#else\n\t\t goto error;\n#endif\n\t\t else\n\t\t type = TYPE_CHAR;\n\t\t break;\n#ifdef HAVE_WINT_T\n\t\tcase 'C':\n\t\t type = TYPE_WIDE_CHAR;\n\t\t c = 'c';\n\t\t break;\n#endif\n\t\tcase 's':\n\t\t if (flags >= 8)\n#ifdef HAVE_WCHAR_T\n\t\t type = TYPE_WIDE_STRING;\n#else\n\t\t goto error;\n#endif\n\t\t else\n\t\t type = TYPE_STRING;\n\t\t break;\n#ifdef HAVE_WCHAR_T\n\t\tcase 'S':\n\t\t type = TYPE_WIDE_STRING;\n\t\t c = 's';\n\t\t break;\n#endif\n\t\tcase 'p':\n\t\t type = TYPE_POINTER;\n\t\t break;\n\t\tcase 'n':\n#ifdef HAVE_LONG_LONG\n\t\t if (flags >= 16 || (flags & 4))\n\t\t type = TYPE_COUNT_LONGLONGINT_POINTER;\n\t\t else\n#endif\n\t\t if (flags >= 8)\n\t\t type = TYPE_COUNT_LONGINT_POINTER;\n\t\t else if (flags & 2)\n\t\t type = TYPE_COUNT_SCHAR_POINTER;\n\t\t else if (flags & 1)\n\t\t type = TYPE_COUNT_SHORT_POINTER;\n\t\t else\n\t\t type = TYPE_COUNT_INT_POINTER;\n\t\t break;\n\t\tcase '%':\n\t\t type = TYPE_NONE;\n\t\t break;\n\t\tdefault:\n\t\t /* Unknown conversion character. */\n\t\t goto error;\n\t\t}\n\t }\n\n\t if (type != TYPE_NONE)\n\t {\n\t\tdp->arg_index = arg_index;\n\t\tif (dp->arg_index == ARG_NONE)\n\t\t {\n\t\t dp->arg_index = arg_posn++;\n\t\t if (dp->arg_index == ARG_NONE)\n\t\t /* arg_posn wrapped around. */\n\t\t goto error;\n\t\t }\n\t\tREGISTER_ARG (dp->arg_index, type);\n\t }\n\t dp->conversion = c;\n\t dp->dir_end = cp;\n\t }\n\n\t d->count++;\n\t if (d->count >= d_allocated)\n\t {\n\t size_t memory_size;\n\t DIRECTIVE *memory;\n\n\t d_allocated = xtimes (d_allocated, 2);\n\t memory_size = xtimes (d_allocated, sizeof (DIRECTIVE));\n\t if (size_overflow_p (memory_size))\n\t\t/* Overflow, would lead to out of memory. */\n\t\tgoto error;\n\t memory = realloc (d->dir, memory_size);\n\t if (memory == NULL)\n\t\t/* Out of memory. */\n\t\tgoto error;\n\t d->dir = memory;\n\t }\n\t}\n }\n d->dir[d->count].dir_start = cp;\n\n d->max_width_length = max_width_length;\n d->max_precision_length = max_precision_length;\n return 0;\n\nerror:\n if (a->arg)\n free (a->arg);\n if (d->dir)\n free (d->dir);\n return -1;\n}\n\n#undef DIRECTIVES\n#undef DIRECTIVE\n#undef CHAR_T\n#undef PRINTF_PARSE\n\n\n#ifndef IN_LIBINTL\n# include \n#endif\n\n\n#include \t/* memcpy(), strlen() */\n#include \t/* errno */\n#include \t/* DBL_MAX_EXP, LDBL_MAX_EXP */\n\n/* Some systems, like OSF/1 4.0 and Woe32, don't have EOVERFLOW. */\n#ifndef EOVERFLOW\n# define EOVERFLOW E2BIG\n#endif\n\n#ifdef HAVE_WCHAR_T\n# ifdef HAVE_WCSLEN\n# define local_wcslen wcslen\n# else\n /* Solaris 2.5.1 has wcslen() in a separate library libw.so. To avoid\n a dependency towards this library, here is a local substitute.\n Define this substitute only once, even if this file is included\n twice in the same compilation unit. */\n# ifndef local_wcslen_defined\n# define local_wcslen_defined 1\nstatic size_t local_wcslen (const wchar_t *s) {\n const wchar_t *ptr;\n\n for (ptr = s; *ptr != (wchar_t) 0; ptr++)\n ;\n return ptr - s;\n}\n# endif\n# endif\n#endif\n\n#if WIDE_CHAR_VERSION\n# define VASNPRINTF vasnwprintf\n# define CHAR_T wchar_t\n# define DIRECTIVE wchar_t_directive\n# define DIRECTIVES wchar_t_directives\n# define PRINTF_PARSE wprintf_parse\n# define USE_SNPRINTF 1\n# if HAVE_DECL__SNWPRINTF\n /* On Windows, the function swprintf() has a different signature than\n on Unix; we use the _snwprintf() function instead. */\n# define SNPRINTF _snwprintf\n# else\n /* Unix. */\n# define SNPRINTF swprintf\n# endif\n#else\n# define VASNPRINTF vasnprintf\n# define CHAR_T char\n# define DIRECTIVE char_directive\n# define DIRECTIVES char_directives\n# define PRINTF_PARSE printf_parse\n# define USE_SNPRINTF (HAVE_DECL__SNPRINTF || HAVE_SNPRINTF)\n# if HAVE_DECL__SNPRINTF\n /* Windows. */\n# define SNPRINTF _snprintf\n# else\n /* Unix. */\n# define SNPRINTF snprintf\n# endif\n#endif\n\nCHAR_T * VASNPRINTF (CHAR_T *resultbuf, size_t *lengthp, const CHAR_T *format, va_list args) {\n DIRECTIVES d;\n arguments a;\n\n if (PRINTF_PARSE (format, &d, &a) < 0) {\n errno = EINVAL;\n return NULL;\n }\n\n#define CLEANUP() \\\n free (d.dir);\t\t\t\t\t\t\t\t\\\n if (a.arg)\t\t\t\t\t\t\t\t\\\n free (a.arg);\n\n if (printf_fetchargs (args, &a) < 0) {\n CLEANUP ();\n errno = EINVAL;\n return NULL;\n }\n\n {\n size_t buf_neededlength;\n CHAR_T *buf;\n CHAR_T *buf_malloced;\n const CHAR_T *cp;\n size_t i;\n DIRECTIVE *dp;\n /* Output string accumulator. */\n CHAR_T *result;\n size_t allocated;\n size_t length;\n\n /* Allocate a small buffer that will hold a directive passed to\n sprintf or snprintf. */\n buf_neededlength =\n xsum4 (7, d.max_width_length, d.max_precision_length, 6);\n#if HAVE_ALLOCA\n if (buf_neededlength < 4000 / sizeof (CHAR_T)) {\n buf = (CHAR_T *) alloca (buf_neededlength * sizeof (CHAR_T));\n buf_malloced = NULL;\n } else\n#endif\n {\n size_t buf_memsize = xtimes (buf_neededlength, sizeof (CHAR_T));\n if (size_overflow_p (buf_memsize))\n goto out_of_memory_1;\n buf = (CHAR_T *) malloc (buf_memsize);\n if (buf == NULL)\n goto out_of_memory_1;\n buf_malloced = buf;\n }\n\n if (resultbuf != NULL) {\n result = resultbuf;\n allocated = *lengthp;\n } else {\n result = NULL;\n allocated = 0;\n }\n length = 0;\n /* Invariants:\n result is either == resultbuf or == NULL or malloc-allocated.\n If length > 0, then result != NULL. */\n\n /* Ensures that allocated >= needed. Aborts through a jump to\n out_of_memory if needed is SIZE_MAX or otherwise too big. */\n#define ENSURE_ALLOCATION(needed) \\\n if ((needed) > allocated)\t\t\t\t\t\t \\\n {\t\t\t\t\t\t\t\t\t \\\n\tsize_t memory_size;\t\t\t\t\t\t \\\n\tCHAR_T *memory;\t\t\t\t\t\t\t \\\n\t\t\t\t\t\t\t\t\t \\\n\tallocated = (allocated > 0 ? xtimes (allocated, 2) : 12);\t \\\n\tif ((needed) > allocated)\t\t\t\t\t \\\n\t allocated = (needed);\t\t\t\t\t\t \\\n\tmemory_size = xtimes (allocated, sizeof (CHAR_T));\t\t \\\n\tif (size_overflow_p (memory_size))\t\t\t\t \\\n\t goto out_of_memory;\t\t\t\t\t\t \\\n\tif (result == resultbuf || result == NULL)\t\t\t \\\n\t memory = (CHAR_T *) malloc (memory_size);\t\t\t \\\n\telse\t\t\t\t\t\t\t\t \\\n\t memory = (CHAR_T *) realloc (result, memory_size);\t\t \\\n\tif (memory == NULL)\t\t\t\t\t\t \\\n\t goto out_of_memory;\t\t\t\t\t\t \\\n\tif (result == resultbuf && length > 0)\t\t\t\t \\\n\t memcpy (memory, result, length * sizeof (CHAR_T));\t\t \\\n\tresult = memory;\t\t\t\t\t\t \\\n }\n\n for (cp = format, i = 0, dp = &d.dir[0]; ; cp = dp->dir_end, i++, dp++)\n {\n\tif (cp != dp->dir_start)\n\t {\n\t size_t n = dp->dir_start - cp;\n\t size_t augmented_length = xsum (length, n);\n\n\t ENSURE_ALLOCATION (augmented_length);\n\t memcpy (result + length, cp, n * sizeof (CHAR_T));\n\t length = augmented_length;\n\t }\n\tif (i == d.count)\n\t break;\n\n\t/* Execute a single directive. */\n\tif (dp->conversion == '%') {\n\t size_t augmented_length;\n\n\t if (!(dp->arg_index == ARG_NONE))\n\t abort ();\n\t augmented_length = xsum (length, 1);\n\t ENSURE_ALLOCATION (augmented_length);\n\t result[length] = '%';\n\t length = augmented_length;\n\t} else {\n\t if (!(dp->arg_index != ARG_NONE))\n\t abort ();\n\n\t if (dp->conversion == 'n') {\n\t\tswitch (a.arg[dp->arg_index].type) {\n\t\t case TYPE_COUNT_SCHAR_POINTER:\n\t\t *a.arg[dp->arg_index].a.a_count_schar_pointer = length;\n\t\t break;\n\t\t case TYPE_COUNT_SHORT_POINTER:\n\t\t *a.arg[dp->arg_index].a.a_count_short_pointer = length;\n\t\t break;\n\t\t case TYPE_COUNT_INT_POINTER:\n\t\t *a.arg[dp->arg_index].a.a_count_int_pointer = length;\n\t\t break;\n\t\t case TYPE_COUNT_LONGINT_POINTER:\n\t\t *a.arg[dp->arg_index].a.a_count_longint_pointer = length;\n\t\t break;\n#ifdef HAVE_LONG_LONG\n\t\t case TYPE_COUNT_LONGLONGINT_POINTER:\n\t\t *a.arg[dp->arg_index].a.a_count_longlongint_pointer = length;\n\t\t break;\n#endif\n\t\t default:\n\t\t abort ();\n\t\t }\n\t }\n\t else {\n\t\targ_type type = a.arg[dp->arg_index].type;\n\t\tCHAR_T *p;\n\t\tunsigned int prefix_count;\n\t\tint prefixes[2];\n#if !USE_SNPRINTF\n\t\tsize_t tmp_length;\n\t\tCHAR_T tmpbuf[700];\n\t\tCHAR_T *tmp;\n\n\t\t/* Allocate a temporary buffer of sufficient size for calling\n\t\t sprintf. */\n\t\t{\n\t\t size_t width;\n\t\t size_t precision;\n\n\t\t width = 0;\n\t\t if (dp->width_start != dp->width_end) {\n\t\t if (dp->width_arg_index != ARG_NONE) {\n int arg;\n\n if (!(a.arg[dp->width_arg_index].type == TYPE_INT))\n abort ();\n arg = a.arg[dp->width_arg_index].a.a_int;\n width = (arg < 0 ? (unsigned int) (-arg) : arg);\n\t\t\t } else {\n const CHAR_T *digitp = dp->width_start;\n\n do\n width = xsum (xtimes (width, 10), *digitp++ - '0');\n while (digitp != dp->width_end);\n\t\t\t }\n\t\t }\n\n\t\t precision = 6;\n\t\t if (dp->precision_start != dp->precision_end) {\n\t\t if (dp->precision_arg_index != ARG_NONE) {\n int arg;\n\n if (!(a.arg[dp->precision_arg_index].type == TYPE_INT))\n abort ();\n arg = a.arg[dp->precision_arg_index].a.a_int;\n precision = (arg < 0 ? 0 : arg);\n\t\t } else {\n const CHAR_T *digitp = dp->precision_start + 1;\n\n precision = 0;\n while (digitp != dp->precision_end)\n precision = xsum (xtimes (precision, 10), *digitp++ - '0');\n }\n\t\t }\n\n\t\t switch (dp->conversion) {\n\n\t\t case 'd': case 'i': case 'u':\n# ifdef HAVE_LONG_LONG\n\t\t if (type == TYPE_LONGLONGINT || type == TYPE_ULONGLONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long long) * CHAR_BIT\n\t\t\t\t\t * 0.30103 /* binary -> decimal */\n\t\t\t\t\t * 2 /* estimate for FLAG_GROUP */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t else\n# endif\n\t\t if (type == TYPE_LONGINT || type == TYPE_ULONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long) * CHAR_BIT\n\t\t\t\t\t * 0.30103 /* binary -> decimal */\n\t\t\t\t\t * 2 /* estimate for FLAG_GROUP */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t else\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned int) * CHAR_BIT\n\t\t\t\t\t * 0.30103 /* binary -> decimal */\n\t\t\t\t\t * 2 /* estimate for FLAG_GROUP */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t break;\n\n\t\t case 'o':\n# ifdef HAVE_LONG_LONG\n\t\t if (type == TYPE_LONGLONGINT || type == TYPE_ULONGLONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long long) * CHAR_BIT\n\t\t\t\t\t * 0.333334 /* binary -> octal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t else\n# endif\n\t\t if (type == TYPE_LONGINT || type == TYPE_ULONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long) * CHAR_BIT\n\t\t\t\t\t * 0.333334 /* binary -> octal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t else\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned int) * CHAR_BIT\n\t\t\t\t\t * 0.333334 /* binary -> octal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 1; /* account for leading sign */\n\t\t break;\n\n\t\t case 'x': case 'X':\n# ifdef HAVE_LONG_LONG\n\t\t if (type == TYPE_LONGLONGINT || type == TYPE_ULONGLONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long long) * CHAR_BIT\n\t\t\t\t\t * 0.25 /* binary -> hexadecimal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 2; /* account for leading sign or alternate form */\n\t\t else\n# endif\n\t\t if (type == TYPE_LONGINT || type == TYPE_ULONGINT)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned long) * CHAR_BIT\n\t\t\t\t\t * 0.25 /* binary -> hexadecimal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 2; /* account for leading sign or alternate form */\n\t\t else\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (sizeof (unsigned int) * CHAR_BIT\n\t\t\t\t\t * 0.25 /* binary -> hexadecimal */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 2; /* account for leading sign or alternate form */\n\t\t break;\n\n\t\t case 'f': case 'F':\n# ifdef HAVE_LONG_DOUBLE\n\t\t if (type == TYPE_LONGDOUBLE)\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (LDBL_MAX_EXP\n\t\t\t\t\t * 0.30103 /* binary -> decimal */\n\t\t\t\t\t * 2 /* estimate for FLAG_GROUP */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 10; /* sign, decimal point etc. */\n\t\t else\n# endif\n\t\t\ttmp_length =\n\t\t\t (unsigned int) (DBL_MAX_EXP\n\t\t\t\t\t * 0.30103 /* binary -> decimal */\n\t\t\t\t\t * 2 /* estimate for FLAG_GROUP */\n\t\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 10; /* sign, decimal point etc. */\n\t\t tmp_length = xsum (tmp_length, precision);\n\t\t break;\n\n\t\t case 'e': case 'E': case 'g': case 'G':\n\t\t case 'a': case 'A':\n\t\t tmp_length =\n\t\t\t12; /* sign, decimal point, exponent etc. */\n\t\t tmp_length = xsum (tmp_length, precision);\n\t\t break;\n\n\t\t case 'c':\n# if defined HAVE_WINT_T && !WIDE_CHAR_VERSION\n\t\t if (type == TYPE_WIDE_CHAR)\n\t\t\ttmp_length = MB_CUR_MAX;\n\t\t else\n# endif\n\t\t\ttmp_length = 1;\n\t\t break;\n\n\t\t case 's':\n# ifdef HAVE_WCHAR_T\n\t\t if (type == TYPE_WIDE_STRING) {\n\t\t\t tmp_length =\n\t\t\t local_wcslen (a.arg[dp->arg_index].a.a_wide_string);\n\n# if !WIDE_CHAR_VERSION\n\t\t\t tmp_length = xtimes (tmp_length, MB_CUR_MAX);\n# endif\n\t\t\t}\n\t\t else\n# endif\n\t\t\ttmp_length = strlen (a.arg[dp->arg_index].a.a_string);\n\t\t break;\n\n\t\t case 'p':\n\t\t tmp_length =\n\t\t\t(unsigned int) (sizeof (void *) * CHAR_BIT\n\t\t\t\t\t* 0.25 /* binary -> hexadecimal */\n\t\t\t\t )\n\t\t\t + 1 /* turn floor into ceil */\n\t\t\t + 2; /* account for leading 0x */\n\t\t break;\n\n\t\t default:\n\t\t abort ();\n\t\t }\n\n\t\t if (tmp_length < width)\n\t\t tmp_length = width;\n\n\t\t tmp_length = xsum (tmp_length, 1); /* account for trailing NUL */\n\t\t}\n\n\t\tif (tmp_length <= sizeof (tmpbuf) / sizeof (CHAR_T))\n\t\t tmp = tmpbuf;\n\t\telse {\n\t\t size_t tmp_memsize = xtimes (tmp_length, sizeof (CHAR_T));\n\n\t\t if (size_overflow_p (tmp_memsize))\n\t\t /* Overflow, would lead to out of memory. */\n\t\t goto out_of_memory;\n\t\t tmp = (CHAR_T *) malloc (tmp_memsize);\n\t\t if (tmp == NULL)\n\t\t /* Out of memory. */\n\t\t goto out_of_memory;\n\t\t}\n#endif\n\n\t\t/* Construct the format string for calling snprintf or\n\t\t sprintf. */\n\t\tp = buf;\n\t\t*p++ = '%';\n\t\tif (dp->flags & FLAG_GROUP)\n\t\t *p++ = '\\'';\n\t\tif (dp->flags & FLAG_LEFT)\n\t\t *p++ = '-';\n\t\tif (dp->flags & FLAG_SHOWSIGN)\n\t\t *p++ = '+';\n\t\tif (dp->flags & FLAG_SPACE)\n\t\t *p++ = ' ';\n\t\tif (dp->flags & FLAG_ALT)\n\t\t *p++ = '#';\n\t\tif (dp->flags & FLAG_ZERO)\n\t\t *p++ = '0';\n\t\tif (dp->width_start != dp->width_end) {\n\t\t size_t n = dp->width_end - dp->width_start;\n\t\t memcpy (p, dp->width_start, n * sizeof (CHAR_T));\n\t\t p += n;\n\t\t}\n\t\tif (dp->precision_start != dp->precision_end) {\n\t\t size_t n = dp->precision_end - dp->precision_start;\n\t\t memcpy (p, dp->precision_start, n * sizeof (CHAR_T));\n\t\t p += n;\n\t\t}\n\n\t\tswitch (type) {\n#ifdef HAVE_LONG_LONG\n\t\t case TYPE_LONGLONGINT:\n\t\t case TYPE_ULONGLONGINT:\n\t\t *p++ = 'l';\n\t\t /*FALLTHROUGH*/\n#endif\n\t\t case TYPE_LONGINT:\n\t\t case TYPE_ULONGINT:\n#ifdef HAVE_WINT_T\n\t\t case TYPE_WIDE_CHAR:\n#endif\n#ifdef HAVE_WCHAR_T\n\t\t case TYPE_WIDE_STRING:\n#endif\n\t\t *p++ = 'l';\n\t\t break;\n#ifdef HAVE_LONG_DOUBLE\n\t\t case TYPE_LONGDOUBLE:\n\t\t *p++ = 'L';\n\t\t break;\n#endif\n\t\t default:\n\t\t break;\n\t\t }\n\t\t*p = dp->conversion;\n#if USE_SNPRINTF\n\t\tp[1] = '%';\n\t\tp[2] = 'n';\n\t\tp[3] = '\\0';\n#else\n\t\tp[1] = '\\0';\n#endif\n\n\t\t/* Construct the arguments for calling snprintf or sprintf. */\n\t\tprefix_count = 0;\n\t\tif (dp->width_arg_index != ARG_NONE) {\n\t\t if (!(a.arg[dp->width_arg_index].type == TYPE_INT))\n\t\t abort ();\n\t\t prefixes[prefix_count++] = a.arg[dp->width_arg_index].a.a_int;\n\t }\n\t\tif (dp->precision_arg_index != ARG_NONE) {\n\t\t if (!(a.arg[dp->precision_arg_index].type == TYPE_INT))\n\t\t abort ();\n\t\t prefixes[prefix_count++] = a.arg[dp->precision_arg_index].a.a_int;\n\t }\n\n#if USE_SNPRINTF\n\t\t/* Prepare checking whether snprintf returns the count\n\t\t via %n. */\n\t\tENSURE_ALLOCATION (xsum (length, 1));\n\t\tresult[length] = '\\0';\n#endif\n\n\t\tfor (;;) {\n\t\t size_t maxlen = allocated - length;\n\t\t int count = -1;\n\n#if USE_SNPRINTF\n\t\t int retcount = 0;\n# define SNPRINTF_BUF(arg) \\\n\t\t switch (prefix_count)\t\t\t\t \\\n\t\t {\t\t\t\t\t\t\t \\\n\t\t case 0:\t\t\t\t\t\t \\\n\t\t\tretcount = SNPRINTF (result + length, maxlen, buf, \\\n\t\t\t\t\t arg, &count);\t\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t case 1:\t\t\t\t\t\t \\\n\t\t\tretcount = SNPRINTF (result + length, maxlen, buf, \\\n\t\t\t\t\t prefixes[0], arg, &count);\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t case 2:\t\t\t\t\t\t \\\n\t\t\tretcount = SNPRINTF (result + length, maxlen, buf, \\\n\t\t\t\t\t prefixes[0], prefixes[1], arg, \\\n\t\t\t\t\t &count);\t\t\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t default:\t\t\t\t\t\t \\\n\t\t\tabort ();\t\t\t\t\t \\\n\t\t }\n#else\n# define SNPRINTF_BUF(arg) \\\n\t\t switch (prefix_count)\t\t\t\t \\\n\t\t {\t\t\t\t\t\t\t \\\n\t\t case 0:\t\t\t\t\t\t \\\n\t\t\tcount = sprintf (tmp, buf, arg);\t\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t case 1:\t\t\t\t\t\t \\\n\t\t\tcount = sprintf (tmp, buf, prefixes[0], arg);\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t case 2:\t\t\t\t\t\t \\\n\t\t\tcount = sprintf (tmp, buf, prefixes[0], prefixes[1],\\\n\t\t\t\t\t arg);\t\t\t\t \\\n\t\t\tbreak;\t\t\t\t\t\t \\\n\t\t default:\t\t\t\t\t\t \\\n\t\t\tabort ();\t\t\t\t\t \\\n\t\t }\n#endif\n\n\t\t switch (type) {\n\t\t case TYPE_SCHAR: {\n\t\t\t int arg = a.arg[dp->arg_index].a.a_schar;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_UCHAR: {\n\t\t\t unsigned int arg = a.arg[dp->arg_index].a.a_uchar;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_SHORT: {\n\t\t\t int arg = a.arg[dp->arg_index].a.a_short;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_USHORT: {\n\t\t\t unsigned int arg = a.arg[dp->arg_index].a.a_ushort;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_INT: {\n\t\t\t int arg = a.arg[dp->arg_index].a.a_int;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_UINT: {\n\t\t\t unsigned int arg = a.arg[dp->arg_index].a.a_uint;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_LONGINT: {\n\t\t\t long int arg = a.arg[dp->arg_index].a.a_longint;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_ULONGINT: {\n\t\t\t unsigned long int arg = a.arg[dp->arg_index].a.a_ulongint;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#ifdef HAVE_LONG_LONG\n\t\t case TYPE_LONGLONGINT: {\n\t\t\t long long int arg = a.arg[dp->arg_index].a.a_longlongint;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t case TYPE_ULONGLONGINT: {\n\t\t\t unsigned long long int arg = a.arg[dp->arg_index].a.a_ulonglongint;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#endif\n\t\t case TYPE_DOUBLE: {\n\t\t\t double arg = a.arg[dp->arg_index].a.a_double;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#ifdef HAVE_LONG_DOUBLE\n\t\t case TYPE_LONGDOUBLE: {\n\t\t\t long double arg = a.arg[dp->arg_index].a.a_longdouble;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#endif\n\t\t case TYPE_CHAR: {\n\t\t\t int arg = a.arg[dp->arg_index].a.a_char;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#ifdef HAVE_WINT_T\n\t\t case TYPE_WIDE_CHAR: {\n\t\t\t wint_t arg = a.arg[dp->arg_index].a.a_wide_char;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#endif\n\t\t case TYPE_STRING: {\n\t\t\t const char *arg = a.arg[dp->arg_index].a.a_string;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#ifdef HAVE_WCHAR_T\n\t\t case TYPE_WIDE_STRING: {\n\t\t\t const wchar_t *arg = a.arg[dp->arg_index].a.a_wide_string;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n#endif\n\t\t case TYPE_POINTER: {\n\t\t\t void *arg = a.arg[dp->arg_index].a.a_pointer;\n\t\t\t SNPRINTF_BUF (arg);\n\t\t\t}\n\t\t\tbreak;\n\t\t default:\n\t\t\tabort ();\n\t }\n\n#if USE_SNPRINTF\n\t\t /* Portability: Not all implementations of snprintf()\n\t\t are ISO C 99 compliant. Determine the number of\n\t\t bytes that snprintf() has produced or would have\n\t\t produced. */\n\t\t if (count >= 0) {\n /* Verify that snprintf() has NUL-terminated its\n result. */\n if (count < maxlen && result[length + count] != '\\0')\n abort ();\n /* Portability hack. */\n if (retcount > count)\n count = retcount;\n\t\t } else {\n /* snprintf() doesn't understand the '%n'\n directive. */\n if (p[1] != '\\0') {\n /* Don't use the '%n' directive; instead, look\n at the snprintf() return value. */\n p[1] = '\\0';\n continue;\n } else {\n /* Look at the snprintf() return value. */\n if (retcount < 0) {\n /* HP-UX 10.20 snprintf() is doubly deficient:\n It doesn't understand the '%n' directive,\n *and* it returns -1 (rather than the length\n that would have been required) when the\n buffer is too small. */\n size_t bigger_need =\n xsum (xtimes (allocated, 2), 12);\n ENSURE_ALLOCATION (bigger_need);\n continue;\n } else\n count = retcount;\n }\n\t\t }\n#endif\n\n\t\t /* Attempt to handle failure. */\n\t\t if (count < 0) {\n if (!(result == resultbuf || result == NULL))\n free (result);\n if (buf_malloced != NULL)\n free (buf_malloced);\n CLEANUP ();\n errno = EINVAL;\n return NULL;\n\t\t }\n\n#if !USE_SNPRINTF\n\t\t if (count >= tmp_length)\n\t\t /* tmp_length was incorrectly calculated - fix the\n\t\t\t code above! */\n\t\t abort ();\n#endif\n\n\t\t /* Make room for the result. */\n\t\t if (count >= maxlen)\n\t\t {\n\t\t\t/* Need at least count bytes. But allocate\n\t\t\t proportionally, to avoid looping eternally if\n\t\t\t snprintf() reports a too small count. */\n\t\t\tsize_t n =\n\t\t\t xmax (xsum (length, count), xtimes (allocated, 2));\n\n\t\t\tENSURE_ALLOCATION (n);\n#if USE_SNPRINTF\n\t\t\tcontinue;\n#endif\n\t\t }\n\n#if USE_SNPRINTF\n\t\t /* The snprintf() result did fit. */\n#else\n\t\t /* Append the sprintf() result. */\n\t\t memcpy (result + length, tmp, count * sizeof (CHAR_T));\n\t\t if (tmp != tmpbuf)\n\t\t free (tmp);\n#endif\n\n\t\t length += count;\n\t\t break;\n\t\t }\n\t }\n\t }\n }\n\n /* Add the final NUL. */\n ENSURE_ALLOCATION (xsum (length, 1));\n result[length] = '\\0';\n\n if (result != resultbuf && length + 1 < allocated)\n {\n\t/* Shrink the allocated memory if possible. */\n\tCHAR_T *memory;\n\n\tmemory = (CHAR_T *) realloc (result, (length + 1) * sizeof (CHAR_T));\n\tif (memory != NULL)\n\t result = memory;\n }\n\n if (buf_malloced != NULL)\n free (buf_malloced);\n CLEANUP ();\n *lengthp = length;\n if (length > INT_MAX)\n goto length_overflow;\n return result;\n\n length_overflow:\n /* We could produce such a big string, but its length doesn't fit into\n an 'int'. POSIX says that snprintf() fails with errno = EOVERFLOW in\n this case. */\n if (result != resultbuf)\n free (result);\n errno = EOVERFLOW;\n return NULL;\n\n out_of_memory:\n if (!(result == resultbuf || result == NULL))\n free (result);\n if (buf_malloced != NULL)\n free (buf_malloced);\n out_of_memory_1:\n CLEANUP ();\n errno = ENOMEM;\n return NULL;\n }\n}\n\n#undef SNPRINTF\n#undef USE_SNPRINTF\n#undef PRINTF_PARSE\n#undef DIRECTIVES\n#undef DIRECTIVE\n#undef CHAR_T\n#undef VASNPRINTF\n\nint vasprintf (char **resultp, const char *format, va_list args) {\n size_t length;\n char *result = vasnprintf (NULL, &length, format, args);\n if (result == NULL)\n return -1;\n\n *resultp = result;\n /* Return the number of resulting bytes, excluding the trailing NUL.\n If it wouldn't fit in an 'int', vasnprintf() would have returned NULL\n and set errno to EOVERFLOW. */\n return length;\n}\n#endif\n /** \\endcond */ //Doxygen ignore.\n" }, { "path": "cmd/Makefile.am", "content": "#The programs\nbin_PROGRAMS = \\\n\tapop_text_to_db \\\n\tapop_db_to_crosstab \\\n\tapop_plot_query\n\nAM_CFLAGS = \\\n\t-I $(top_srcdir) \\\n\t$(GSL_CFLAGS)\n\nLDADD = \\\n\t$(top_builddir)/libapophenia.la \\\n\t$(GSL_LIBS)\n\n" }, { "path": "cmd/apop_db_to_crosstab.c", "content": "/** \\file \nCommand line utility to convert a three-column table to a crosstab.*/\n\n/*Copyright (c) 2005--2007, 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\nint main(int argc, char **argv){\n int c;\n char verbose=0;\n char const *msg=\"Usage: %s [opts] dbname table_name rows columns data\\n\"\n\"\\n\"\n\"A command-line wrapper for the apop_db_to_crosstab function.\\n\"\n\"See Apophenia's online documentation for that function for details and tricks.\\n\"\n\"The default for the data column is a count [count(*)]\\n\"\n\"The column is optional; leave it out if you want a single-dimensional crosstab.\\n\"\n\"If you need a non-default data column but want a 1-D crosstab, use 1 as your column.\\n\"\n\"\\n\"\n\" -d\\tdelimiter (default: )\\n\"\n\" -v\\tverbose: prints status info on stderr\\n\"\n\" -v -v\\tvery verbose: also print queries executed on stderr\\n\"\n\" -h\\tdisplay this help and exit\\n\"\n\"\\n\";\n\n\tapop_opts.verbose=0; //so don't print queries until -v -v.\n\n\twhile ((c = getopt (argc, argv, \"d:f:hv-\")) != -1)\n\t\tif (c=='d') strcpy(apop_opts.output_delimiter,optarg);\n else if (c=='h'||c=='-') {printf(msg, argv[0]); exit(0);}\n else if (c=='v') {\n verbose++;\n apop_opts.verbose++;\n }\n\n Apop_stopif(optind+2 > argc, return 1, 0, \"I need at least two arguments past the options: database table [optional rowcol] [optional columncol] [optional datacol]\");\n _Bool no_rowcol = optind+2 > argc;\n _Bool no_columncol = optind+3 > argc;\n _Bool no_datacol = optind+4 > argc;\n char *rowcol = no_rowcol ? \"1\" : argv[optind+2];\n char *colcol = no_columncol ? \"1\" : argv[optind+3];\n char *datacol = no_datacol ? NULL: argv[optind+4];\n if (verbose){\n fprintf(stderr, \"database:%s\\ntable: %s\\nrow col: %s\\ncol col:%s%s%s\\n---------\\n\",\n argv[optind], argv[optind +1], rowcol, colcol,\n no_datacol ?\"\":\"\\ndata col:\", datacol);\n }\n\tapop_db_open(argv[optind]);\n\tapop_data *m = apop_db_to_crosstab(argv[optind +1], rowcol, colcol, datacol);\n\tapop_data_print(m);\n}\n" }, { "path": "cmd/apop_plot_query.c", "content": "/** \\file \n Command line utility to take in a query and produce a plot of its output via Gnuplot.\n\nCopyright (c) 2006--2007 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\n\n/** This convenience function will take in a \\c gsl_vector of data and put out a histogram, ready to pipe to Gnuplot.\n\n\\param data A \\c gsl_vector holding the data. Do not pre-sort or bin; this function does that for you via apop_data_to_bins.\n\\param bin_count The number of bins in the output histogram (if you send zero, I set this to \\f$\\sqrt(N)\\f$, where \\f$N\\f$ is the length of the vector.)\n\\param with The method for Gnuplot's plotting routine. Default is \\c \"boxes\", so the gnuplot call will read plot '-' with boxes. The \\c \"lines\" option is also popular, and you can add extra terms if desired, like \"boxes linetype 3\".\n*/\nvoid plot_histogram(gsl_vector *data, FILE *f, size_t bin_count, char *with){\n Apop_stopif(!data, return, 0, \"Input vector is NULL.\");\n if (!with) with=\"impulses\";\n apop_data vector_as_data = (apop_data){.vector=data};\n apop_data *histodata = apop_data_to_bins(&vector_as_data, .bin_count=bin_count, .close_top_bin='y');\n apop_data_sort(apop_data_pmf_compress(histodata));\n apop_data_free(histodata->more); //the binspec.\n\n fprintf(f, \"set key off\t;\\n\"\n \"plot '-' with %s\\n\", with);\n apop_data_print(histodata, .output_pipe=f);\n fprintf(f, \"e\\n\");\n\n fflush(f);\n apop_data_free(histodata);\n}\n\n\nchar *plot_type = NULL;\nint histobins = 0;\nint histoplotting = 0;\n\nFILE *open_output(char *outfile, int sf){\n FILE *f;\n if (sf && !strcmp (outfile, \"-\"))\n return stdout;\n if (sf && outfile){\n f = fopen(outfile, \"w\");\n Apop_stopif(!f, exit(0), 0, \"Trouble opening %s.\", outfile);\n return f;\n }\n f = popen(\"`which gnuplot` -persist\", \"w\");\n Apop_stopif(!f, exit(0), 0, \"Trouble opening %s.\", \"gnuplot\");\n return f;\n}\n\nchar *read_query(char *infile){\n char in[1000];\n char *q = malloc(10);\n q[0] = '\\0';\n FILE *inf = fopen(infile, \"r\");\n Apop_stopif(!inf, exit(0), 0, \"Trouble opening %s.\\n\", infile);\n while(fgets(in, 1000, inf)){\n q = realloc(q, strlen(q) + strlen(in) + 4);\n sprintf(q, \"%s%s\", q, in);\n }\n sprintf(q, \"%s;\\n\", q);\n fclose(inf);\n return q;\n}\n\ngsl_matrix *query(char *d, char *q, int no_plot){\n\tapop_db_open(d);\n apop_data *result = apop_query_to_data(\"%s\", q);\n\tapop_db_close(0);\n Apop_stopif(!result, exit(2), 0, \"Your query returned a blank table. Quitting.\");\n Apop_stopif(result->error, exit(2), 0, \"Error running your query. Quitting.\");\n if (no_plot){\n apop_data_show(result);\n exit(0);\n }\n return result->matrix;\n}\n\nvoid print_out(FILE *f, char *outfile, gsl_matrix *m){\n if (!histoplotting){\n fprintf(f,\"plot '-' with %s\\n\", plot_type);\n\t apop_matrix_print(m, NULL, .output_type='p', .output_pipe=f);\n } else {\n Apop_col_v(&(apop_data){.matrix=m}, 0, v);\n plot_histogram(v, f, histobins, NULL);\n }\n if (outfile) fclose(f);\n}\n\nint main(int argc, char **argv){\n int c;\n char *q = NULL,\n *d = NULL,\n *outfile = NULL;\n int sf = 0,\n no_plot = 0;\n\n const char* msg= \"Usage: %s [opts] dbname query\\n\"\n\"\\n\"\n\"Runs a query, and pipes the output directly to gnuplot. Use -f to dump to stdout or a file.\\n\"\n\" -d\\tdatabase to use (mandatory)\\n\"\n\" -q\\tquery to run (mandatory or use -Q)\\n\"\n\" -Q\\tfile from which to read the query\\t\\t\\n\"\n\" -n\\tno plot: just run the query and display results to stdout\\t\\t\\n\"\n\" -t\\tplot type (points, bars, ...) (default: \\\"lines\\\")\\n\"\n\" -H\\tplot histogram with this many bins (e.g., -H100) (to let the system auto-select bin sizes, use -H0)\\n\"\n\" -f\\tfile to dump to. If -f- then use stdout (default: pipe to Gnuplot)\\n\"\n\" -h\\tdisplay this help and exit\\n\"\n\"\\n\";\n\n\tApop_stopif(argc<2, return 1, 0, msg, argv[0]);\n\twhile ((c = getopt (argc, argv, \"ad:f:hH:nQ:q:st:-\")) != -1)\n\t if (c=='f'){\n outfile = strdup(optarg);\n sf++;\n } else if (c=='H'){\n histoplotting = 1;\n histobins = atoi(optarg);\n }\n else if (c=='h'||c=='-') {\n printf(msg, argv[0]);\n\t\t\treturn 0;\n\t\t}\n else if (c=='d') d = strdup(optarg);\n else if (c=='n') no_plot ++;\n else if (c=='Q') q = read_query(optarg);\n else if (c=='q') q = strdup(optarg);\n else if (c=='t') plot_type = strdup(optarg);\n\n if (optind == argc -2){\n d = argv[optind];\n q = argv[optind+1];\n } else if (optind == argc-1)\n q = argv[optind];\n\n Apop_stopif(!q, return 1, 0, \"I need a query specified with -q.\\n\");\n\n if (!plot_type) plot_type = strdup(\"lines\");\n\n FILE *f = open_output(outfile, sf);\n gsl_matrix *m = query(d, q, no_plot);\n print_out(f, outfile, m);\n}\n" }, { "path": "cmd/apop_text_to_db.c", "content": "/** \\file \n A command line script to read a text file into a database.\n\nCopyright (c) 2006--2007, 2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n#include \"apop_internal.h\"\n#include \n\nint *break_down(char *in){\n int *out = NULL;\n int ctr = 0;\n char *cp = strtok (in, \",\");\n while (cp != NULL) {\n out = realloc(out, sizeof(int)*(ctr+1));\n out[ctr++] = atoi(cp);\n cp = strtok (NULL, \",\");\n }\n return out;\n}\n\nint main(int argc, char **argv){\n int c;\n char *msg;\n int colnames = 'y',\n rownames = 0,\n tab_exists_check = 0;\n char **field_names = NULL;\n\n\tAsprintf(&msg, \"Usage: %s [-d delimiters] text_file table_name dbname\\n\"\n\"\\n\"\n\"If the input text file name is a single dash, -, then read from STDIN.\\n\"\n\"Input must be plain ASCII or UTF-8.\\n\"\n\" -d\\t\\tthe single-character delimiters to use, e.g., -d \\\" ,\\\" or -d \\\"\\\\t\\\" (which you \\n\"\n \" \\t\\t\\twill almost certainly have to write as -d \\\"\\\\\\\\t\\\") (default: \\\"|,\\\\t\\\", meaning \\n\"\n \" \\t\\t\\tthat any of a pipe, comma, or tab will delimit separate entries)\\n\"\n\" -nc\\t\\tdata does not include column names\\n\"\n\" -n regex\\t\\tcase-insensitive regular expression indicating Null values (default: NaN)\\n\"\n\" -m\\t\\tuse a MySQL database (default: SQLite)\\n\"\n\" -f\\t\\tfixed width field ends: -f\\\"3,8,12,17\\\" (first char is one, not zero)\\n\"\n\" -u\\t\\tmysql username\\n\"\n\" -p\\t\\tmysql password\\n\"\n\" -r\\t\\tdata includes row names\\n\"\n\" -v\\t\\tverbosity\\n\"\n\" -N\\t\\ta comma-separated list of column names: -N\\\"apple,banana,carrot,durian\\\"\\n\"\n\" -en\\t\\tif table exists, do nothing and exit\\n\"\n\" -ed\\t\\tif table exists, retain the table, delete all data, refill with the new data (i.e., call 'delete * from your_table')\\n\"\n\" -eo\\t\\tif table exists, overwrite the table from scratch (deleting the previous table entirely)\\n\"\n\" -ea\\t\\tif table exists, append new data to the existing table\\n\"\n\" -h\\t\\tdisplay this help and exit\\n\"\n\"\\n\"\n, argv[0]);\n int * field_list = NULL;\n char if_exists = 'n';\n\n\tif(argc<3){\n\t\tprintf(\"%s\", msg);\n\t\treturn 0;\n\t}\n\twhile ((c = getopt (argc, argv, \"n:d:e:f:hmp:ru:vN:O\")) != -1)\n if (c=='n') {\n if (optarg[0]=='c') colnames='n';\n else apop_opts.nan_string = optarg;\n }\n\t\telse if (c=='N') {\n apop_data *field_name_data;\n apop_regex(optarg, \" *([^,]*[^ ]) *(,|$) *\", &field_name_data);\n Apop_stopif(!field_name_data, return 1, 0, \"'%s' should be a \"\n \"comma-delimited list of field names, but I had trouble \"\n \"parsing it as such.\", optarg);\n apop_data_transpose(field_name_data);\n field_names = field_name_data->text[0];\n }\n else if (c=='d') strcpy(apop_opts.input_delimiters, optarg);\n\t\telse if (c=='f') field_list = break_down(optarg);\n\t\telse if (c=='h') {printf(\"%s\", msg); return 0;}\n\t\telse if (c=='m') apop_opts.db_engine = 'm';\n\t\telse if (c=='u') strcpy(apop_opts.db_user, optarg);\n\t\telse if (c=='p') strcpy(apop_opts.db_pass, optarg);\n\t\telse if (c=='r') rownames++;\n\t\telse if (c=='v') apop_opts.verbose=2;\n\t\telse if (c=='O') tab_exists_check++; //deprecated as of December 2013.\n\t\telse if (c=='e') {\n if (optarg[0]=='n') if_exists='n'; //the default anyway.\n else if (optarg[0]=='d') if_exists='d';\n else if (optarg[0]=='a') if_exists='a';\n else if (optarg[0]=='o') {if_exists='o';\n tab_exists_check++;\n }\n\n }\n\tapop_db_open(argv[optind + 2]);\n if (tab_exists_check) apop_table_exists(argv[optind+1],1);\n apop_query(\"begin\");\n\tapop_text_to_db(argv[optind], argv[optind+1], rownames, colnames, field_names, .field_ends=field_list, .if_table_exists=if_exists);\n apop_query(\"commit\");\n}\n" }, { "path": "configure", "content": "#!/bin/bash\n\nversion=1.0\nworkdir=apophenia-$version\n\necho This configure script uses Autotools to generate a build directory, $workdir.\n\nif [ ! -x `which autoreconf` ]\nthen\necho \"I couldn't find and execute autoreconf, part of the Autotools suite. See README for details.\"\nelse\n\necho \"Don't forget: Apophenia always depends on the GNU Scientific Library and SQLite3 (_and_ their -dev or -devel packages) to build and run.\"\nmkdir -p $workdir/m4\ncp -r cmd docs eg install/{acinclude.m4,apophenia.pc.in,configure.ac,Makefile.am,COPYING,Readme-pkg,rpm.spec,apophenia.map} model tests transform README ChangeLog $workdir\n\n#Debian maintainers didn't want this in the makefile.\ncp *.c $workdir && rm $workdir/*.m4.c\ncp apop_internal.h $workdir\nm4 -P install/prep_variadics.m4 apop.m4.h > $workdir/apop.h.in\nfor i in *.m4.c; do\n m4 -P install/prep_variadics.m4 $i > $workdir/`basename $i .m4.c`.c\ndone\n\n#mkdir -p $workdir/tests/apophenia\n#cp *.h $workdir/tests/apophenia\ncd $workdir\n\n\n#Autoconf requirements:\ncp ChangeLog NEWS\necho \"Ben Klemens (fluffmail@f-m.fm)\" > AUTHORS\n\nautoreconf -f -i\nautoreconf -i\n./configure -C\n#make dist\nmake distcheck\n\necho\necho ---------------------\necho\necho \"OK, built. From the $workdir directory, you can run: make && sudo make install\"\n\nfi #end `else' block of test for Autotools.\n" }, { "path": "docs/Makefile.am", "content": "\n#The Doxygen documentation, which you'll have to call yourself (via make doc).\n\nGVZDOT = /usr/bin/dot\n\ndefault:\n\n## adhoc\nhtml man: doc\n\napophenia_CSOURCES = \\\n\t$(top_srcdir)/model/*.c \\\n\t$(top_srcdir)/transform/*.c\n\techo #$(wildcard $(top_srcdir)/model/*.c) \\\n\techo #$(wildcard $(top_srcdir)/transform/*.c)\n\napophenia_DOTS = \\\n\tstructs.dot\n\napophenia_IMAGES = \\\n\tflake.gif \\\n\tsearch.gif \\\n\tright.png \\\n\tdown.png\n\napophenia_IMAGES_EXTRA = \\\n\ttriangle.png \\\n\tmodel.png\n\napophenia_JS = \\\n\ttree.js\n\napophenia_IMAGES_GENERATED = \\\n\tstructs.png\n\nmodel_doc.h: $(apophenia_CSOURCES)\n\tcat $^ | awk -f $(top_srcdir)/docs/make_model_doc.awk > $@\n\ndoc: documentation.h model_doc.h $(apophenia_IMAGES) $(apophenia_JS) $(apophenia_IMAGES_GENERATED)\n\t$(MKDIR_P) include\n\tsed -e 's/__attribute.*;/;/;s/extern //' $(top_builddir)/apop.h > include/apop.h\n\tdoxygen doxygen.conf\n\tfor f in $(apophenia_IMAGES) $(apophenia_JS) ; do \\\n\t\ttest -f $(top_srcdir)/docs/$$f && cp $(top_srcdir)/docs/$$f html/ ;\\\n\tdone\n\tcp $(apophenia_IMAGES_GENERATED) html/\n\tsed -i -f $(top_srcdir)/docs/edit_outline.sed html/index.html html/outline.html\n\tsed -i -f $(top_srcdir)/docs/edit_group.sed html/group__models.html\n\tsed -i -f $(top_srcdir)/docs/edit_width.sed html/*.html\n\t$(abs_top_srcdir)/docs/adjust\n\ndoc-clean:\n\t-rm -rf include html latex man\n\ninstall-html-local: doc\n\tcp -prd html $(DESTDIR)$(docdir)\n\nmaintainer-clean-local: doc-clean\n\nCLEANFILES = \\\n\tmissing_model_parts\n\nMAINTAINERCLEANFILES = \\\n\tmodel_doc.h \\\n\tdoxygen.log\n\nEXTRA_DIST = \\\n\tadjust \\\n\tmake_model_doc.awk \\\n\t$(apophenia_DOTS) \\\n\t$(apophenia_IMAGES) \\\n\t$(apophenia_IMAGES_EXTRA) \\\n\t$(apophenia_JS) \\\n\tedit_outline.sed edit_globals.sed edit_group.sed edit_width.sed \\\n\tapop_data_fig.html head.html foot.html \\\n\ttypical.css \\\n\tdocumentation.h\n\n%.png : %.dot\n\t$(GVZDOT) -Tpng < $< > $@\n" }, { "path": "docs/adjust", "content": "#Use GNU sed to make modifications to the LaTeX version before producing a PDF\n\n#But first, some HTML tweaks:\n# Background color of side pane should match rest of document\nsed -i \"/background-color/ s|:.*|: #FFFFDE;|\" html/navtree.css\nsed -i \"/nav_h.png/ d\" html/navtree.css\n\n# We no longer live in an ASCII-only world.\nsed -i 's/pdflatex/xelatex/' latex/Makefile\n\n#PDF: a 170pp document shouldn't be typeset in sans serif\nsed -i \"/familydefault.*sfdefault/ d\" latex/refman.tex\n\n#No fancyheaders\nsed -i -e \"/fancy/d\" -e '/footrulewidth/d' -e \"/Generated by Doxygen/ d\" latex/refman.tex\n\n##Set title, for headers and such.\n#sed '/begin.document/i\n#\\\\title[Apophenia]{Apophenia}' latex/refman.tex\n\n# Nicer lists.\nsed -i '/begin.document/a\\\n\\\\setlength\\\\itemsep{0pt} \\\\renewcommand\\\\labelitemi{$\\\\circ$} \\\\renewcommand\\\\labelitemii{$\\\\circ$} \\\\renewcommand\\\\labelitemiii{$\\\\cdot$}' latex/refman.tex\n\n#I can't make the index look not-stupid\nsed -i '/makeindex/d' latex/refman.tex\nsed -i '/makeindex/d' latex/Makefile\n\n\n#Fix enumerations -> model documentation\nsed -i -f edit_group.sed latex/group__models.tex\n#Rm header list\nsed -i -e '/subsection.*Enumer/d' -e'/begin{DoxyCompactItemize}/,/end{DoxyCompactItemize}/d' latex/group__models.tex\n##sed -i -e 's/\\\\cline{.*}$//' -e's/begin{TabularN*C}.*/begin{tabular}{ll}/' -e's/end{TabularN*C}.*/end{tabular}/' latex/group__models.tex\n#sed -i -e's/\\(TabularC}\\){2}/\\1{3}/' latex/group__models.tex\n\n#redundant with the struct defns.\nfdline=$((`grep -n 'subsection.Function Documentation.' latex/group__all__public.tex|sed 's/:.*//'`-1))\nsed -i \"/subsection.Typedef Documentation./,$fdline d\" latex/group__all__public.tex\n\n#Unclutter the ToC by using section*=omit from ToC\nsed -i -e 's/subsection{Detailed Description}/subsection*{Detailed Description}/' -e 's/subsection{Field Documentation}/subsection*{Field Documentation}/' latex/*tex\n\n#Bare references to subpages\n#sed -i -e's|^\\\\hyperlink.*hypertarget.*subsection.*label{[^}]*}$||' latex/*tex\nsed -i -e's|^\\\\hyperlink{[^}]*}{[^}]*}$||' latex/*tex\n\n#the hyperlink insertion into sample code sometimes creates needless newlines with a\n#six-space indent.\nsed -i '$!N;s/\\n \\(\\\\hyperlink\\)/\\1/;P;D' latex/*tex\n\n#code looks too small to me. Go one step bigger.\nsed -i -e 's/scriptsize/footnotesize/' latex/doxygen.sty\n\n#paragraphs have a lot of space after the header; tighten\n#sed -i -e'/DoxyParagraph/,+15 s|\\(end{list}\\)|\\1\\\\vspace{-10pt}|' latex/doxygen.sty\n\n#offset description labels by a few cm\nsed -i '$a\\\n\\\\renewcommand{\\\\descriptionlabel}[1]{\\\\hspace{-1.5cm}\\\\emph{#1}}' latex/doxygen.sty\n\nsed -i -e 's/xtab\\*{\\\\linewidth}/xtab/g' -e 's/xtab\\(ular\\|\\)/supertabular/g' -e 's/\\\\tablefirsthead.*}//' -e '/\\\\par%/d' latex/doxygen.sty\n#sed latex/doxygen.sty\n\n\n\n#Want search box in header\nsed -i -e '/position: *absolute/d' html/search/search.css\nsed -i -e 's/display: *block/display: inline/' html/search/search.css\n#magnifying glass is hard-coded, and span=\"left\" covers left and middle of search box\nsed -i -e '/mag_sel.png/,+3d' html/*html\n\n#100% of the search results are doubled. Can't work out why.\nsed -i -e 's/\\],\\[[^]]\\+\\]/]/' html/search/*js\n\n" }, { "path": "docs/apop_data_fig.html", "content": "\n\n\n\n\n
RownameVector Matrix TextWeights
\n\n\n\n\n\n\n\n \n
\"Steven\"
\"Sandra\"
\"Joe\"\n
\n		\n\n\n\n\n\n\n\n\n\n \n
Outcome
1
0
1
\n		\n\n\n\n\n\n\n\n\n\n \n
Age Weight (kg) Height (cm)
32 65 175
41 61 165
40 73 181
\n		\n\n\n\n\n\n\n\n\n\n \n
Sex State
Male Alaska\n
Female Alabama
Male Alabama
\n		\n\n\n\n\n\n\n\n\n \n
1
3.2
2.4
\n
\n" }, { "path": "docs/apop_data_fig.tex", "content": "\\begin{tabular}{ccccc}\nRowname& Vector& Matrix& Text& Weights\\\\\n\\fbox{\n\\begin{tabular}{c}\n\\phantom{hi}\\\\\n\"Steven\"\\\\\n\"Sandra\"\\\\\n\"Joe\"\n\\end{tabular}\n}\n&\n\\fbox{\n\\begin{tabular}{c}\n{\\bf Outcome}\\\\\n1\\\\\n0\\\\\n1\n\\end{tabular}\n}\n&\n\\fbox{\n\\begin{tabular}{ccc}\n{\\bf Age} & {\\bf Weight (kg)} & {\\bf Height (cm)}\\\\\n32 & 65 & 175\\\\\n41 & 61 & 165\\\\\n40 & 73 & 181\n\\end{tabular}\n}\n&\n\\fbox{\n\\begin{tabular}{ccc}\n{\\bf Sex} & {\\bf State} \\\\\nMale & Alaska\\\\\nFemale & Alabama\\\\\nMale & Alabama\n\\end{tabular}\n}\n&\n\\fbox{\n\\begin{tabular}{ccc}\n\\phantom{hi}\\\\\n1\\\\\n3.2\\\\\n2.4\n\\end{tabular}\n}\n\\end{tabular}\n" }, { "path": "docs/documentation.h", "content": "/* Apophenia's narrative documentation\nCopyright (c) 2005--2013 by Ben Klemens. Licensed under the GPLv2; see COPYING. */\n\n/** \\mainpage Welcome\n\nApophenia is an open statistical library for working with data sets and statistical\nmodels. It provides functions on the same level as those of the typical stats package\n(such as OLS, Probit, or singular value decomposition) but gives the user more\nflexibility to be creative in model-building. The core functions are written in C,\nbut experience has shown them to be easy to bind to in Python/Julia/Perl/Ruby/&c.\n\nIt is written to scale well, to comfortably work with gigabyte data sets, million-step simulations, or\ncomputationally-intensive agent-based models.\n\nThe goods \n\nThe library has been growing and improving since 2005, and has been downloaded well over 1e4 times. To date, it has over two hundred functions and macros to facilitate statistical computing, such as:\n\n\\li OLS and family, discrete choice models like Probit and Logit, kernel density estimators, and other common models.\n\\li Functions for transforming models (like Normal \\f$\\rightarrow\\f$ truncated Normal)\n and combining models (produce the cross-product of that truncated Normal with three others,\n or use Bayesian updating to combine that cross-product prior with an OLS likelihood\n to produce a posterior distribution over the OLS parameters).\n\\li Database querying and maintenance utilities.\n\\li Data manipulation tools for splitting, stacking, sorting, and otherwise shunting data sets.\n\\li Moments, percentiles, and other basic stats utilities.\n\\li \\f$t\\f$-tests, \\f$F\\f$-tests, et cetera.\n\\li Several optimization methods available for your own new models.\n\\li It does not re-implement basic matrix operations or build yet another database\nengine. Instead, it builds upon the excellent GNU\nScientific and SQLite libraries. MySQL/mariaDB is also supported.\n\nFor the full list of macros, functions, and prebuilt models, check the index.\n\nDownload Apophenia here.\n\nOr, see the \\ref setup page for detailed setup instructions,\nincluding how to use your package manager to install the Debian or Homebrew package.\n\nThe documentation\n\nTo start off, have a look at this \\ref gentle \"Gentle Introduction\" to the library.\n\nThe outline gives a more detailed narrative.\n\nThe index lists every function in the\nlibrary, with detailed reference\ninformation. Notice that the header to every page has a link to the outline and the index.\n\nTo really go in depth, download or pick up a copy of Modeling with Data, which discusses general\nmethods for doing statistics in C with the GSL and SQLite, as well as Apophenia\nitself. A Useful\nAlgebraic System of Statistical Models (PDF) discusses some of the theoretical\nstructures underlying the library.\n\nThere is a wiki with some convenience\nfunctions, tips, and so on.\n\nNotable features \nMuch of what Apophenia does can be done in any typical statistics package. The \\ref\napop_data structure is much like an R data frame, for example, and there is nothing special\nabout being able to invert a matrix or take the product of two matrices with a single\nfunction call (\\ref apop_matrix_inverse and \\ref apop_dot, respectively). \nEven more advanced features like Loess smoothing (\\ref apop_loess) and the Fisher Exact\nTest (\\ref apop_test_fisher_exact) are not especially Apophenia-specific. But here are\nsome things that are noteworthy.\n\n \\li It's a C library! You can build applications using Apophenia for the data-processing\nback-end of your program, and not worry about the overhead associated with scripting\nlanguages. For example, it is currently used in production for certain aspects of\nprocessing for the U.S. Census Bureau's American Community Survey. And the numeric\nroutines in your favorite scripting language typically have a back-end in plain C;\nperhaps Apophenia can facilitate writing your next one.\n \\li The \\ref apop_model object allows for consistent treatment of distributions, regressions,\nsimulations, machine learning models, and who knows what other sorts of models you can\ndream up. By transforming and combining existing models, it is easy to build complex\nmodels from simple sub-models.\n \\li For example, the \\ref apop_update function does Bayesian updating on any two\nwell-formed models. If they are on the table of conjugates, that is correctly\nhandled, and if they are not, an appropriate variant of MCMC produces an empirical\ndistribution. The output is yet another model, from which you can make random draws,\nor which you can use as a prior for another round of Bayesian updating. Outside of\nBayesian updating, the \\ref apop_model_metropolis function is good for approximating\nother complex models.\n \\li The maximum likelihood system combines several subsystems into one\nform: it will do a few flavors of conjugate gradient search, Nelder-Mead Simplex,\nNewton's Method, or Simulated Annealing. You pick the method by a setting attached to\nyour model. If you want to use a method that requires derivatives and you don't have\na closed-form derivative, the ML subsystem will estimate a numerical gradient for\nyou. If you would like to do EM-style maximization (all but the first parameter are\nfixed, that parameter is optimized, then all but the second parameter are fixed, that\nparameter is optimized, ..., looping through dimensions until the change in objective\nacross cycles is less than eps), add a settings group specifying the\ntolerance at which the cycle should stop: Apop_settings_add_group(your_model,\napop_mle, .dim_cycle_tolerance=eps).\n \\li The Iterative Proportional Fitting algorithm, \\ref apop_rake, is best-in-breed,\ndesigned to handle large, sparse matrices.\n\n\n\nContribute! \n\n\\li Develop a new model object.\n\\li Contribute your favorite statistical routine.\n\\li Package Apophenia into an RPM, portage, cygwin package.\n\\li Report bugs or suggest features.\n\\li Write bindings for your preferred language. For example, here are \na Perl wrapper and early versions\nof a Julia wrapper and an R wrapper which you could expand upon.\n\nIf you're interested, write to the maintainer (Ben Klemens), or join the\nGitHub project.\n*/\n\n/** \\page eg Some examples\nHere are a few pieces of sample code for testing your installation or to give you a\nsense of what code with Apophenia's tools looks like.\n\n Two data streams\n\nThe sample program here is intended to show how one would integrate Apophenia into an\nexisting program. For example, say that you are running a simulation of two different\ntreatments, or say that two sensors are posting data at regular intervals. The goal is to\ngather the data in an organized form, and then ask questions of the resulting data set.\nBelow, a thousand draws are made from the two processes and put into a database. Then,\nthe data is pulled out, some simple statistics are compiled, and the data is written to\na text file for inspection outside of the program. This program will compile cleanly\nwith the sample \\ref makefile.\n\n\\include draw_to_db.c\n\n Run a regression\n\nSee \\ref gentle for an example of loading a data set and running a simple regression.\n\n A sequence of t-tests\n\nIn \\ref mapply \"The section on map/apply\", a new \\f$t\\f$-test on every row, with all\noperations acting on entire rows rather than individual data points:\n\n\\include t_test_by_rows.c\n\nIn the documentation for \\ref apop_query_to_text, a program to list all the tables in an SQLite database.\n\\include ls_tables.c\n\nMarginal distribution\n\nA demonstration of fixing parameters to create a marginal distribution, via \\ref apop_model_fix_params\n\\include fix_params.c\n*/\n\n\n/** \\page setup Setting up\n\\section cast The supporting cast \nTo use Apophenia, you will need to have a working C compiler, the GSL (v1.7 or higher) and SQLite installed. mySQL/mariaDB is optional.\n\n\\li Some readers may be unfamiliar with modern package managers and common methods for setting up a C development environment; see \nAppendix O of Modeling with Data for an introduction.\n\n\\li Other pages in this documentation have a few more notes for \\ref windows \"Windows\" users, including \\ref mingw users.\n\n\\li Install the basics using your package manager. E.g., try\n\n\\code\nsudo apt-get install make gcc libgsl0-dev libsqlite3-dev\n\\endcode\n\nor\n\n\\code\nsudo yum install make gcc gsl-devel libsqlite3x-devel\n\\endcode\n\n\\li If you use a Debian-based system (including Ubuntu), try the version in Debian's Stretch distribution:\n\n\\code\n#Add Stretch to your sources list\nsudo sed -i '$a\ndeb http://httpredir.debian.org/debian stretch main\ndeb-src http://httpredir.debian.org/debian stretch main' /etc/apt/sources.list\nsudo apt-get update\n\n#Get Apophenia\nsudo apt-get install apophenia-bin apophenia-doc libapophenia2 libapophenia2-dev libapophenia2-dbg\n\n\n#Optional: remove Stretch from your sources list.\nsudo sed -i -e '$d' -e '$d' /etc/apt/sources.list\n\\endcode\n\nThanks to Jerome Benoit for building the package and bringing the library up to Debian standards.\n\n\\li Mac users with Homebrew, try\n\\code \nbrew install homebrew/science/apophenia\n\\endcode\n\nThanks to bdobyns for the Brew script.\n\n\\li Or, Download Apophenia here.\nOnce you have the library downloaded, compile it using\n\n\\code\ntar xvzf apop*tgz && cd apophenia-0.999\n./configure && make && make check && sudo make install\n\\endcode\n\nIf you decide not to keep the library on your system, run sudo make uninstall\nfrom the source directory to remove it.\n\n\\li If you need to install packages in your home directory because you don't have root\npermissions, see the \\ref notroot page.\n\n\\li A \\ref makefile will help immensely when you want to compile your program.\n\n\\li You can verify that your setup works by trying some \\ref eg \"sample programs\".\n\n\\subpage notroot\n\n\\subpage makefile\n\n\\subpage windows\n\n\n\\li Those who would like to work on a cutting-edge copy of the source code\ncan get the latest version by cutting and pasting the following onto\nthe command line. If you follow this route, be sure to read the development README in the\napophenia directory this command will create.\n\n\\code\ngit clone https://github.com/b-k/apophenia.git\n\\endcode\n\n\n\n*/\n\n/** \\page windows Windows\n\n\\ref mingw users, see that page.\n\nIf you have a choice, Cygwin is strongly recommended. The setup program is\nvery self-explanatory. As a warning, it will probably take up >300MB on\nyour system. You should install at least the following programs:\n\\li autoconf/automake\n\\li binutils\n\\li gcc\n\\li gdb\n\\li gnuplot -- for plotting data\n\\li groff -- needed for the man program, below\n\\li gsl -- the engine that powers Apophenia\n\\li less -- to read text files\n\\li libtool -- needed for compiling programs\n\\li make\n\\li man -- for reading help files\n\\li more -- not as good as less but still good to have\n\\li sqlite3 -- a simple database engine, a requisite for Apophenia\n\nIf you are missing anything else, the program will probably tell you.\nThe following are not necessary but are good to have on hand as long as you are going to be using Unix and programming.\n\n\\li git -- to partake in the versioning system\n\\li emacs -- steep learning curve, but people love it\n\\li ghostscript (for reading .ps/.pdf files)\n\\li openssh -- needed for git\n\\li perl, python, ruby -- these are other languages that you might also be interested in\n\\li tetex -- write up your documentation using the nicest-looking formatter around\n\\li X11 -- a windowing system\n\nX-Window will give you a nicer environment in which to work. After you start Cygwin, type startx to bring up a more usable, nice-looking terminal (and the ability to do a few thousand other things which are beyond the scope of this documentation). Once you have Cygwin installed and a good terminal running, you can follow along with the remainder of the discussion without modification.\n\nSome older versions of Cygwin have a \\c search.h file which doesn't include the function lsearch(). If this is the case on your system, you will have to update your Cygwin installation.\n\nFinally, windows compilers often spit out lines like:\n\\code\nInfo: resolving _gsl_rng_taus by linking to __imp__gsl_rng_taus (auto-import)\n\\endcode\nThese lines are indeed just information, and not errors. Feel free to ignore them.\n\n[Thanks to Andrew Felton and Derrick Higgins for their Cygwin debugging efforts.]\n\n\\subpage mingw\n*/\n\n/** \\page notroot Not root? \nIf you aren't root, then the common procedure for installing a library is to create a subdirectory in your home directory in which to install packages. The key is the --prefix addition to the ./configure command.\n\\code\nexport MY_LIBS = myroot #choose a directory name to be created in your home directory.\nmkdir $HOME/$MY_LIBS\n\n# From Apophenia's package directory:\n./configure --prefix $HOME/$MY_LIBS\nmake\nmake install #Now you don't have to be root.\n\n# Adjust your paths so the compiler and the OS can find the library.\n# These are environment variables, and they are usually set in the \n# shell's startup files. I assume you are using bash here.\n\necho \"export PATH=$HOME/$MY_LIBS/include:\\$PATH\" >> ~/.bashrc\necho \"export CPATH=$HOME/$MY_LIBS/include:\\$CPATH\" >> ~/.bashrc\necho \"export LIBRARY_PATH=$HOME/$MY_LIBS:\\$LIBRARY_PATH\" >> ~/.bashrc\necho \"export LD_LIBRARY_PATH=$HOME/$MY_LIBS:\\$LD_LIBRARY_PATH\" >> ~/.bashrc\n\\endcode\n\nOnce you have created this local root directory, you can use it to install as many new libraries as desired, and your paths will already be set up to find them.\n*/\n\n\n/** \\page makefile Makefile\n \nInstead of giving lengthy compiler commands at the command prompt, you can use a Makefile to do most of the work. How to:\n\\li Copy and paste the following into a file named \\c makefile.\n\\li Change the first line to the name of your program (e.g., if you have written census.c, then the first line will read PROGNAME=census). \n\\li If your program has multiple .c files, add a corresponding .o to the currently blank objects variable, e.g. objects=sample2.o sample3.o\n\\li One you have a Makefile in the directory, simply type make at the command prompt to generate the executable.\n\n \\code\nPROGNAME = your_program_name_here\nobjects =\nCFLAGS = -g -Wall -O3\nLDLIBS = -lapophenia -lgsl -lgslcblas -lsqlite3\n\n$(PROGNAME): $(objects)\n\\endcode\n\n\\li If your system has \\c pkg-config, then you can use it for a slightly more robust and readable makefile. Replace the above C and link flags with:\n\\code\nCFLAGS = -g -Wall `pkg-config --cflags apophenia` -O3\nLDLIBS = `pkg-config --libs apophenia`\n\\endcode\nThe \\c pkg-config program will then fill in the appropriate directories and libraries. Pkg-config knows Apophenia depends on the GSL and database libraries, so you need only list the most-dependent library.\n\n\\li The -O3 flag is optional, asking the compiler to run its highest level of optimization (for speed).\n\n\\li GCC users may need the --std=gnu99 or --std=gnu11 flag to use post-1989 C standards.\n\n\\li Order matters in the linking list: the files a package depends on should be listed after the package. E.g., since sample.c depends on Apophenia, gcc sample.c -lapophenia will work, while gcc -lapophenia sample.c is likely to give you errors. Similarly, list -lapophenia before -lgsl, which comes before -lgslcblas.\n\n*/\n\n/** \\page designated Designated initializers\n\nFunctions so marked in this documentation use standard C designated initializers and compound literals to allow you to omit, call by name, or change the order of inputs. The following examples are all equivalent.\n\nThe standard format:\n\\code\napop_text_to_db(\"infile.txt\", \"intable\", 0, 1, NULL);\n\\endcode\n\nOmitted arguments are left at their default vaules:\n\\code\napop_text_to_db(\"infile.txt\", \"intable\");\n\\endcode\n\nYou can use the variable's name, if you forget its ordering:\n\\code\napop_text_to_db(\"infile.txt\", \"intable\", .has_col_name=1, .has_row_name=0);\n\\endcode\n\nIf an un-named element follows a named element, then that value is given to the next variable in the standard ordering:\n\\code\napop_text_to_db(\"infile.txt\", \"intable\", .has_col_name=1, NULL);\n\\endcode\n\n\\li There may be cases where you can not use this form (it relies on a macro, which\nmay not be available). You can always call the underlying function directly, by adding\n\\c _base to the name and giving all arguments:\n\n\\code\napop_text_to_db_base(\"infile.txt\", \"intable\", 0, 1, NULL);\n\\endcode\n\n\\li If one of the optional elements is an RNG and you do not provide one, I use one\nfrom \\ref apop_rng_get_thread.\n*/\n\n\n/** \\page preliminaries Getting started\n\nIf you are entirely new to Apophenia, \\ref gentle \"have a look at the Gentle Introduction here\".\n\nAs well as the information in this outline, there is a separate page covering the details of \n \\ref setup \"setting up a computing environment\" and another page with \\ref eg \"some sample code\" for your perusal.\n\n\\subpage gentle\n\n\\subpage setup\n\n\\subpage eg\n\n\\subpage refstatusext\n*/\n\n/** \\page refstatusext References and extensions\n\n\\section mwd The book version\n\nApophenia co-evolved with Modeling with Data: Tools and Techniques for Statistical Computing. You can read about the book, or download a free PDF copy of the full text, at modelingwithdata.org.\n\nAs with many computer programs, the preferred manner of citing Apophenia is to cite its related book.\nHere is a BibTeX-formatted entry giving the relevant information:\n\n\\code \n@book{klemens:modeling,\n title = \"Modeling with Data: Tools and Techniques for Statistical Computing\",\n author=\"Ben Klemens\",\n year=2008,\n publisher=\"Princeton University Press\"\n}\n\\endcode\n\nThe rationale for the \\ref apop_model struct, based on an algebraic system of models, is detailed in a U.S. Census Bureau research report:\n\n\\code \n@techreport{klemens:algebra,\n title = \"A Useful Algebraic System of Statistical Models\",\n author=\"Ben Klemens\",\n month=jul,\n year=2014,\n institution=\"U.S.\\ Census Bureau\",\n number=\"06\"\n}\n\\endcode\n\n\\section ext How do I write extensions?\n\nThe system is written to not require a registration or initialization step to add a new\nmodel or other such parts. Just write your code and include it like any other\nC code. A new \\ref apop_model has to conform to some rules if it is to play well with\n\\ref apop_estimate, \\ref apop_draw, and so forth. See the notes at \\ref modeldetails.\nOnce your new model or function is working, please post the code or a link to the code\non the Apophenia wiki.\n\n\\subpage c\n\n\\section links Further references\n\nFor your convenience, here are links to some other libraries you are probably using.\n\n\\li The standard C library\n\\li The\nGSL documentation, and its index\n\\li SQL understood by SQLite\n\n\n*/\n\n/** \\page outline An outline of the library\n\nThe narrative in this section goes into greater detail on how to use the \ncomponents of Apophenia. You are encouraged to read \\ref gentle first.\n\nThis overview begins with the \\ref apop_data set, which is the central data structure\nused throughout the system. Section \\ref dbs covers the use of the database interface,\nbecause there are a lot of things that a database will do better than a matrix\nstructure like the \\ref apop_data struct.\n\nSection \\ref modelsec covers statistical models, in the form of the \\ref apop_model structure.\nThis part of the system is built upon the \\ref apop_data set to hold parameters, statistics, data sets, and so on.\n\n\\ref histosec covers probability mass functions, which are statistical models\nbuilt directly around a data set, where the chance of drawing a given observation is\nproportional to how often that observation appears in the source data. There are many\nsituations where one would want to treat a data set as a probability distribution,\nsuch as using \\ref apop_kl_divergence to find the information loss from an observed\ndata set to a theoretical model fit to that data.\n\nSection \\ref testpage covers traditional hypothesis testing, beginning with common\nstatistics that take an \\ref apop_data set or two as input, and continuing on to\ngeneralized hypothesis testing for any \\ref apop_model.\n\nBecause estimation in the \\ref apop_model relies heavily on maximum likelihood\nestimation, Apophenia's optimizer subsystem is extensive. \\ref maxipage offers\nsome additional notes on optimization and how it can be used in non-statistical contexts.\n\n\\subpage dataoverview\n\n\\subpage dbs\n\n\\subpage modelsec\n\n\\subpage histosec\n\n\\subpage testpage\n\n\\subpage maxipage\n\n\\subpage moreasst\n*/\n\n/** \\page c C, SQL and coding utilities\n \nLearning C\n\nModeling with Data has a full tutorial for C, oriented at users of standard stats packages. More nuts-and-bolts tutorials are in abundance. Some people find pointers to be especially difficult; fortunately, there's a claymation cartoon which clarifies everything.\n\nHeader aggregation\n\nThere is only one header. Put\n\\code\n#include \n\\endcode\nat the top of your file, and you're done. Everything declared in that file starts with \\c apop_ or \\c Apop_. It also includes \n\\c assert.h, \\c math.h, \\c signal.h, and \\c string.h.\n\nLinking\n\nYou will need to link to the Apophenia library, which involves adding the -lapophenia flag to your compiler. Apophenia depends on SQLite3 and the GNU Scientific Library (which depends on a BLAS), so you will probably need something like:\n\n\\code\ngcc sample.c -lapophenia -lsqlite3 -lgsl -lgslcblas -o run_me -g -Wall -O3\n\\endcode\n\nYour best bet is to encapsulate this mess in a \\ref makefile \"Makefile\". Even if you are using an IDE and its command-line management tools, see the Makefile page for notes on useful flags.\n\nStandards compliance\n\nTo the best of our abilities, Apophenia complies to the C standard (ISO/IEC\n9899:2011). As well as relying on the GSL and SQLite, it uses some POSIX function calls,\nsuch as \\c strcasecmp and \\c popen.\n\n\\subpage designated\n\n\\section debugging Errors, logging, debugging and stopping\n\nThe \\c error element \n\nThe \\ref apop_data set and the \\ref apop_model both include an element named \\c error. It is normally \\c 0, indicating no (known) error. \n\nFor example, \\ref apop_data_copy detects allocation errors and some circular links\n(when Data->more == Data) and fails in those cases. You could thus use the\nfunction with a form like\n\n\\code\napop_data *d = apop_text_to_data(\"indata\");\napop_data *cp = apop_data_copy(d);\nif (cp->error) {printf(\"Couldn't copy the input data; failing.\\n\"); return 1;}\n\\endcode\n\nThere is sometimes (but not always) benefit to handling specific error codes, which are listed in the documentation of those functions that set the \\c error element. E.g.,\n\n\\code\napop_data *d = apop_text_to_data(\"indata\");\napop_data *cp = apop_data_copy(d);\nif (cp->error == 'a') {printf(\"Couldn't allocate space for the copy; failing.\\n\"); return 1;}\nif (cp->error == 'c') {printf(\"Circular link in the data set; failing.\\n\"); return 2;}\n\\endcode\n\nThe end of Appendix O\nof Modeling with Data offers some GDB macros which can make dealing with\nApophenia from the GDB command line much more pleasant. As discussed below, it\nalso helps to set apop_opts.stop_on_warning='v' or 'w' when running\nunder the debugger.\n\n\n\\section verbsec Verbosity level and logging\n\nThe global variable apop_opts.verbose determines how many notifications and warnings get printed by Apophenia's warning mechanism:\n\n-1: turn off logging, print nothing (ill-advised)
\n0: notify only of failures and clear danger
\n1: warn of technically correct but odd situations that might indicate, e.g., numeric instability
\n2: debugging-type information; print queries
\n3: give me everything, such as the state of the data at each iteration of a loop.\n\nThese levels are of course subjective, but should give you some idea of where to place the\nverbosity level. The default is 1.\n\nThe messages are printed to the \\c FILE* handle at apop_opts.log_file. If\nthis is blank (which happens at startup), then this is set to \\c stderr. This is the\ntypical behavior for a console program. Use\n\n\\code\napop_opts.log_file = fopen(\"mylog\", \"w\");\n\\endcode\n\nto write to the \\c mylog file instead of \\c stderr.\n\nAs well as the error and warning messages, some functions can also print diagnostics,\nusing the \\ref Apop_notify macro. For example, \\ref apop_query and friends will print the\nquery sent to the database engine iff apop_opts.verbose >=2 (which is useful\nwhen building complex queries). The diagnostics attempt to follow\nthe same verbosity scale as the warning messages.\n\n\\section Stopping\n\nBy default, warnings and errors never halt processing. It is up to the calling function\nto decide whether to stop.\n\nWhen running the program under a debugger, this is an annoyance: we want to stop as\nsoon as a problem turns up.\n\nThe global variable apop_opts.stop_on_warning changes when the system halts:\n\n\\c 'n': never halt. If you were using Apophenia to support a user-friendly GUI, for example, you would use this mode.
\nThe default: if the variable is '\\0' (the default), halt on severe errors, continue on all warnings.
\n\\c 'v': If the verbosity level of the warning is such that the warning would print to screen, then halt;\nif the warning message would be filtered out by your verbosity level, continue.
\n\\c 'w': Halt on all errors or warnings, including those below your verbosity threshold.\n\nSee the documentation for individual functions for details on how each reports errors to the caller and the level at which warnings are posted.\n\n\\section Legi Legible output\n\nThe output routines handle four sinks for your output. There is a global variable that\nyou can use for small projects where all data will go to the same place.\n\n\\code \napop_opts.output_type = 'f'; //named file\napop_opts.output_type = 'p'; //a pipe or already-opened file\napop_opts.output_type = 'd'; //the database\n\\endcode\n\nYou can also set the output type, the name of the output file or table, and other options\nvia arguments to individual calls to output functions. See \\ref apop_prep_output for the list of options.\n\nC makes minimal distinction between pipes and files, so you can set a\npipe or file as output and send all output there until further notice:\n\n\\code\napop_opts.output_type = 'p';\napop_opts.output_pipe = popen(\"gnuplot\", \"w\");\napop_plot_lattice(...); //see https://github.com/b-k/Apophenia/wiki/gnuplot_snippets\nfclose(apop_opts.output_pipe);\napop_opts.output_pipe = fopen(\"newfile\", \"w\");\napop_data_print(set1);\nfprintf(apop_opts.output_pipe, \"\\nNow set 2:\\n\");\napop_data_print(set2);\n\\endcode\n\nContinuing the example, you can always override the global data with\na specific request:\n\\code\napop_vector_print(v, \"vectorfile\"); //put vectors in a separate file\napop_matrix_print(m, \"matrix_table\", .output_type = 'd'); //write to the db\napop_matrix_print(m, .output_pipe = stdout); //now show the same matrix on screen\n\\endcode\n\nI will first look to the input file name, then the input pipe, then the\nglobal \\c output_pipe, in that order, to determine to where I should\nwrite. Some combinations (like output type = \\c 'd' and only a pipe) don't\nmake sense, and I'll try to warn you about those. \n\nWhat if you have too much output and would like to use a pager, like \\c less or \\c more?\nIn C and POSIX terminology, you're asking to pipe your output to a paging program. Here is\nthe form:\n\\code\nFILE *lesspipe = popen(\"less\", \"w\");\nassert(lesspipe);\napop_data_print(your_data_set, .output_pipe=lesspipe);\npclose(lesspipe);\n\\endcode\n\\c popen will search your usual program path for \\c less, so you don't have to give a full path.\n\n\\li\\ref apop_data_print \n\\li\\ref apop_matrix_print\n\\li\\ref apop_vector_print\n\n\\section sqlsec About SQL, the syntax for querying databases\n\nFor a reference, your best bet is the Structured Query Language reference for SQLite. For a tutorial; there is an abundance of tutorials online. Here is a nice blog entry about complementaries between SQL and matrix manipulation packages.\n\nApophenia currently supports two database engines: SQLite and mySQL/mariaDB. SQLite is the default, because it is simpler and generally more easygoing than mySQL, and supports in-memory databases.\n\nThe global apop_opts.db_engine is initially \\c NULL, indicating no preference\nfor a database engine. You can explicitly set it:\n\n\\code\napop_opts.db_engine='s' //use SQLite\napop_opts.db_engine='m' //use mySQL/mariaDB\n\\endcode\n\nIf \\c apop_opts.db_engine is still \\c NUL on your first database operation, then I will check\nfor an environment variable APOP_DB_ENGINE, and set \napop_opts.db_engine='m' if it is found and matches (case insensitive) \\c mariadb or \\c mysql.\n\n\\code\nexport APOP_DB_ENGINE=mariadb\napop_text_to_db indata mtab db_for_maria\n\nunset APOP_DB_ENGINE\napop_text_to_db indata stab db_for_sqlite.db\n\\endcode\n\n\nWrite \\ref apop_data sets to the database using \\ref apop_data_print, with .output_type='d'.\n\n\\li Column names are inserted if there are any. If there are, all dots are converted\n to underscores. Otherwise, the columns will be named \\c c1, \\c c2, \\c c3, &c.\n\\li If \\ref apop_opts_type \"apop_opts.db_name_column\" is not blank (the default is\n \"row_name\"), then a so-named column is created, and the row names are placed there.\n\\li If there are weights, they will be the last column of the table, and the column will be named \\c weights.\n\\li If the table does not exist, create it. Use \\ref apop_data_print (data, \"tabname\", .output_type='d', .output_append='w')\n to overwrite an existing table or with .output_append='a' to append.\n Appending is the default. Or,\n call \\ref apop_table_exists (\"tabname\", 'd') to ensure that the table is\n removed ahead of time.\n\n\\li If your data set has zero data (i.e., is just a list of column names or is entirely\n blank), \\ref apop_data_print returns without creating anything in the database.\n\\li Especially if you are using a pre-2007 version of SQLite, there may be a speed\n gain to wrapping the call to this function in a begin/commit pair:\n\n\\code\napop_query(\"begin;\");\napop_data_print(dataset, .output_name=\"dbtab\", .output_type='d');\napop_query(\"commit;\");\n\\endcode\n\n\nFinally, Apophenia provides a few nonstandard SQL functions to facilitate math via database; see \\ref db_moments.\n\n\n\\section threads Threading\n\nApophenia uses OpenMP for threading. You generally do not need to know how OpenMP works\nto use Apophenia, and many points of work will thread without your doing anything.\n\n\\li All functions strive to be thread-safe. Part of how this is achieved is that static\nvariables are marked as thread-local or atomic, as per the C standard. There still\nexist compilers that can't implement thread-local or atomic variables, in which case\nyour safest bet is to set OMP's thread count to one as below (or get a new compiler).\n\n\\li Some functions modify their inputs. It is up to you to use those functions in\na thread-safe manner. The \\ref apop_matrix_realloc handles states and global variables\ncorrectly in a threaded environment, but if you have two threads resizing the same \\c\ngsl_matrix at the same time, you're going to have problems.\n\n\\li There are few compilers that don't support OpenMP. When compiling on such a system all work will be single-threaded.\n\n\\li Set the maximum number of threads to \\c N with the environment variable\n\n\\code\nexport OMP_NUM_THREADS N\n\\endcode\n\nor the C function\n\n\\code\n#include \nomp_set_num_threads(N);\n\\endcode\n\nUse one of these methods with N=1 if you want a single-threaded program.\nYou can return later to using all available threads via omp_set_num_threads(omp_get_num_procs()).\n\n\\li \\ref apop_map and friends distribute their \\c for loop over the input \\ref apop_data\nset across multiple threads. Therefore, be careful to send thread-unsafe functions to\nit only after calling \\c omp_set_num_threads(1).\n\n\\li There are a few functions, like \\ref apop_model_draws, that rely on \\ref apop_map, and\ntherefore also thread by default.\n\n\\li The function \\ref apop_rng_get_thread retrieves a statically-stored RNG specific\nto a given thread. Therefore, if you use that function in the place of a \\c gsl_rng,\nyou can parallelize functions that make random draws.\n\n\\li \\ref apop_rng_get_thread allocates its store of threads using apop_opts.rng_seed,\nthen incrementing that seed by one. You thus probably have threads with seeds 479901,\n479902, 479903, .... [If you have a better way to do it, please feel free to modify the\ncode to implement your improvement and submit a pull request on Github.]\n\nSee this tutorial on C\nthreading if you would like to know more, or are unsure about whether your functions\nare thread-safe or not.\n*/\n\n\n/** \\page dataoverview Data sets\n\nThe \\ref apop_data structure represents a data set. It joins together a \\c gsl_vector,\na \\c gsl_matrix, an \\ref apop_name, and a table of strings. It tries to be lightweight,\nso you can use it everywhere you would use a \\c gsl_matrix or a \\c gsl_vector.\n\nHere is a diagram showing a sample data set with all of the elements in place. Together,\nthey represent a data set where each row is an observation, which includes both numeric\nand text values, and where each row/column may be named.\n\n\\htmlinclude apop_data_fig.html\n\\latexinclude apop_data_fig.tex\n\nIn a regression, the vector would be the dependent variable, and the other columns\n(after factor-izing the text) the independent variables. Or think of the \\ref apop_data\nset as a partitioned matrix, where the vector is column -1, and the first column of\nthe matrix is column zero. Here is some sample code to print the vector and matrix,\nstarting at column -1 (but you can use \\ref apop_data_print to do this).\n\n\\code\nfor (int j = 0; j< data->matrix->size1; j++){\n printf(\"%s\\t\", apop_name_get(data->names, j, 'r'));\n for (int i = -1; i< data->matrix->size2; i++)\n printf(\"%g\\t\", apop_data_get(data, j, i));\n printf(\"\\n\");\n}\n\\endcode\n\nMost functions assume that each row represents one observation, so the data vector,\ndata matrix, and text have the same row count: \\c data->vector->size==data->matrix->size1\nand \\c data->vector->size==*data->textsize. This means that the \\ref apop_name structure\ndoesn't have separate \\c vector_names, \\c row_names, or \\c text_row_names elements:\nthe \\c rownames are assumed to apply for all.\n\nSee below for notes on managing the \\c text element and the row/column names.\n\n\\section pps Pages\n\nThe \\ref apop_data set includes a \\c more pointer, which will typically be \\c NULL,\nbut may point to another \\ref apop_data set. This is intended for a main data set\nand a second or third page with auxiliary information, such as estimated parameters\non the front page and their covariance matrix on page two, or predicted data on the\nfront page and a set of prediction intervals on page two.\n\nThe \\c more pointer is not intended for a linked list for millions of data points. In\nsuch situations, you can often improve efficiency by restructuring your data to use\na single table (perhaps via \\ref apop_data_pack and \\ref apop_data_unpack).\n\nMost functions, such as \\ref apop_data_copy and \\ref apop_data_free, will handle all\nthe pages of information. For example, an optimization search over multi-page parameter\nsets would search the space given by all pages.\n\nPages may also be appended as output or auxiliary information, such as\ncovariances, and an MLE would not search over these elements. Any page with a name in\nXML-ish brackets, such as \\, is considered information about the\ndata, not data itself, and therefore ignored by search routines, missing data routines,\net cetera. This is achieved by a rule in \\ref apop_data_pack and \\ref apop_data_unpack.\n\nHere is a toy example that establishes a baseline data set, adds a page,\nmodifies it, and then later retrieves it.\n\\code\napop_data *d = apop_data_alloc(10, 10, 10); //the base data set, a 10-item vector + 10x10 matrix\napop_data *a_new_page = apop_data_add_page(d, apop_data_alloc(2,2), \"new 2 x 2 page\");\ngsl_vector_set_all(a_new_page->matrix, 3);\n\n//later:\napop_data *retrieved = apop_data_get_page(d, \"new\", 'r'); //'r'=search via regex, not literal match.\napop_data_print(retrieved); //print a 2x2 grid of 3s.\n\\endcode\n\n\\section datafns Functions for using apop_data sets\n\nThere are a great many functions to collate, copy, merge, sort, prune, and otherwise\nmanipulate the \\ref apop_data structure and its components.\n\n\\li\\ref apop_data_add_named_elmt\n\\li\\ref apop_data_copy\n\\li\\ref apop_data_fill\n\\li\\ref apop_data_memcpy\n\\li\\ref apop_data_pack\n\\li\\ref apop_data_rm_columns\n\\li\\ref apop_data_sort\n\\li\\ref apop_data_split\n\\li\\ref apop_data_stack\n\\li\\ref apop_data_transpose : transpose matrices (square or not) and text grids\n\\li\\ref apop_data_unpack\n\\li\\ref apop_matrix_copy\n\\li\\ref apop_matrix_realloc\n\\li\\ref apop_matrix_stack\n\\li\\ref apop_text_set\n\\li\\ref apop_text_paste\n\\li\\ref apop_text_to_data\n\\li\\ref apop_vector_copy\n\\li\\ref apop_vector_fill\n\\li\\ref apop_vector_stack\n\\li\\ref apop_vector_realloc\n\\li\\ref apop_vector_unique_elements\n\nApophenia builds upon the GSL, but it would be inappropriate to redundantly replicate\nthe GSL's documentation here.\nMeanwhile, here are prototypes for a few common functions. The GSL's\nnaming scheme is very consistent, so a simple reminder of the function name may be\nsufficient to indicate how they are used.\n\n\\li gsl_matrix_swap_rows (gsl_matrix * m, size_t i, size_t j)\n\\li gsl_matrix_swap_columns (gsl_matrix * m, size_t i, size_t j)\n\\li gsl_matrix_swap_rowcol (gsl_matrix * m, size_t i, size_t j)\n\\li gsl_matrix_transpose_memcpy (gsl_matrix * dest, const gsl_matrix * src)\n\\li gsl_matrix_transpose (gsl_matrix * m) : square matrices only\n\\li gsl_matrix_set_all (gsl_matrix * m, double x)\n\\li gsl_matrix_set_zero (gsl_matrix * m)\n\\li gsl_matrix_set_identity (gsl_matrix * m)\n\\li gsl_matrix_memcpy (gsl_matrix * dest, const gsl_matrix * src)\n\\li void gsl_vector_set_all (gsl_vector * v, double x)\n\\li void gsl_vector_set_zero (gsl_vector * v)\n\\li int gsl_vector_set_basis (gsl_vector * v, size_t i): set all elements to zero, but set item \\f$i\\f$ to one.\n\\li gsl_vector_reverse (gsl_vector * v): reverse the order of your vector's elements\n\\li gsl_vector_ptr and gsl_matrix_ptr. To increment an element in a vector use, e.g., *gsl_vector_ptr(v, 7) += 3; or (*gsl_vector_ptr(v, 7))++.\n\\li gsl_vector_memcpy (gsl_vector * dest, const gsl_vector * src)\n\n\\subsection readin Reading from text files\n\nThe \\ref apop_text_to_data() function takes in the name of a text file with a grid of data in (comma|tab|pipe|whatever)-delimited format and reads it to a matrix. If there are names in the text file, they are copied in to the data set. See \\ref text_format for the full range and details of what can be read in.\n\nIf you have any columns of text, then you will need to read in via the database: use\n\\ref apop_text_to_db() to convert your text file to a database table, \ndo any database-appropriate cleaning of the input data, then use \\ref\napop_query_to_data() or \\ref apop_query_to_mixed_data() to pull the data to an \\ref apop_data set.\n\n\\subpage text_format\n\n\\section datalloc Alloc/free\n\nYou may not need to use these functions often, given that \\ref apop_query_to_data, \\ref apop_text_to_data, and many transformation functions will auto-allocate \\ref apop_data sets for you.\n\nThe \\ref apop_data_alloc function allocates a vector, a matrix, or both. After this call, the structure will have blank names, \\c NULL \\c text element, and \\c NULL \\c weights. See \\ref names for discussion of filling the names. Use \\ref apop_text_alloc to allocate the \\c text grid. The \\c weights are a simple \\c gsl_vector, so allocate a 100-unit weights vector via allocated_data_set->weights = gsl_vector_alloc(100).\n\nExamples of use can be found throughout the documentation; for example, see \\ref gentle.\n\n\\li\\ref apop_data_alloc\n\\li\\ref apop_data_calloc\n\\li\\ref apop_data_free\n\\li\\ref apop_text_alloc : allocate or resize the text part of an \\ref apop_data set.\n\\li\\ref apop_text_free\n\nSee also:\n\n\\li gsl_matrix * gsl_matrix_alloc (size_t n1, size_t n2)\n\\li gsl_matrix * gsl_matrix_calloc (size_t n1, size_t n2)\n\\li void gsl_matrix_free (gsl_matrix * m)\n\\li gsl_vector * gsl_vector_alloc (size_t n)\n\\li gsl_vector * gsl_vector_calloc (size_t n)\n\\li void gsl_vector_free (gsl_vector * v)\n\n\n\\section gslviews \tUsing views\n\nThere are several macros for the common task of viewing a single row or column of a \\ref\napop_data set.\n\n\\code\napop_data *d = apop_query_to_data(\"select obs1, obs2, obs3 from a_table\");\n\n//Get a column using its name. Note that the generated view, ov, is the\n//last item named in the call to the macro.\nApop_col_t(d, \"obs1\", ov);\ndouble obs1_sum = apop_vector_sum(ov);\n\n//Get row zero of the data set's matrix as a vector; get its sum\ndouble row_zero_sum = apop_vector_sum(Apop_rv(d, 0));\n\n//Get a row or rows as a standalone one-row apop_data set\napop_data_print(Apop_r(d, 0));\n\n//ten rows starting at row 3:\napop_data *d10 = Apop_rs(d, 3, 10);\napop_data_print(d10);\n\n//Column zero's sum\ngsl_vector *cv = Apop_cv(d, 0);\ndouble col_zero_sum = apop_vector_sum(cv);\n//or one one line:\ndouble col_zero_sum = apop_vector_sum(Apop_cv(d, 0));\n\n//Pull a 10x5 submatrix, whose origin element is the (2,3)rd\n//element of the parent data set's matrix\n\ndouble sub_sum = apop_matrix_sum(Apop_subm(d, 2,3, 10,5));\n\\endcode\n\nBecause these macros can be used as arguments to a function, these macros have abbreviated names to save line space.\n\n\\li\\ref Apop_r : get row as one-observation \\ref apop_data set\n\\li\\ref Apop_c : get column as \\ref apop_data set\n\\li\\ref Apop_cv : get column as \\ref gsl_vector\n\\li\\ref Apop_rv : get row as \\ref gsl_vector\n\\li\\ref Apop_cs : get columns as \\ref apop_data set\n\\li\\ref Apop_rs : get rows as \\ref apop_data set\n\\li\\ref Apop_mcv : matrix column as vector\n\\li\\ref Apop_mrv : matrix row as vector\n\\li\\ref Apop_subm : get submatrix of a \\ref gsl_matrix\n\nA second set of macros have a slightly different syntax, taking the name of the object to be declared as the last argument. These can not be used as expressions such as function arguments.\n\n\\li\\ref Apop_col_t\n\\li\\ref Apop_row_t\n\\li\\ref Apop_col_tv\n\\li\\ref Apop_row_tv\n\nThe view is an automatic variable, not a pointer, and therefore disappears at the end\nof the scope in which it is declared. If you want to retain the data after the function\nexits, copy it to another vector:\n\n\\code\nreturn apop_vector_copy(Apop_rv(d, 2)); //return a gsl_vector copy of row 2\n\\endcode\n\nCurly braces always delimit scope, not just at the end of a function. \nWhen program evaluation exits a given block, all variables in that block are\nerased. Here is some sample code that won't work:\n\n\\code\napop_data *outdata;\nif (get_odd){\n outdata = Apop_r(data, 1);\n} else {\n outdata = Apop_r(data, 0);\n}\napop_data_print(outdata); //breaks: outdata points to out-of-scope variables.\n\\endcode\n\nFor this if/then statement, there are two sets of local variables\ngenerated: one for the \\c if block, and one for the \\c else block. By the last line,\nneither exists. You can get around the problem here by making sure to not put the macro\ndeclaring new variables in a block. E.g.:\n\n\\code\napop_data *outdata = Apop_r(data, get_odd ? 1 : 0);\napop_data_print(outdata);\n\\endcode\n\n\n\\section data_set_get Set/get\n\nFirst, some examples:\n\n\\code\napop_data *d = apop_data_alloc(10, 10, 10);\napop_name_add(d->names, \"Zeroth row\", 'r');\napop_name_add(d->names, \"Zeroth col\", 'c');\n\n//set cell at row=8 col=0 to value=27\napop_data_set(d, 8, 0, .val=27);\nassert(apop_data_get(d, 8, .colname=\"Zeroth\") == 27);\ndouble *x = apop_data_ptr(d, .col=7, .rowname=\"Zeroth\");\n*x = 270;\nassert(apop_data_get(d, 0, 7) == 270);\n\n// This is invalid---the value doesn't follow the colname. Use .val=5.\n// apop_data_set(d, .row = 3, .colname=\"Column 8\", 5); \n\n// This is OK, to set (3, 8) to 5:\napop_data_set(d, 3, 8, 5);\n\n\n//apop_data set holding a scalar:\napop_data *s = apop_data_alloc(1);\napop_data_set(s, .val=12);\nassert(apop_data_get(s) == 12);\n\n//apop_data set holding a vector:\napop_data *v = apop_data_alloc(12);\nfor (int i=0; i< 12; i++) apop_data_set(s, i, .val=i*10);\nassert(apop_data_get(s,3) == 30);\n\n//This is a common form from pulling from a list of named scalars, \n//produced via apop_data_add_named_elmt\ndouble AIC = apop_data_get(your_model->info, .rowname=\"AIC\");\n\\endcode\n\n\\li The versions that take a column/row name use \\ref apop_name_find\n for the search; see notes there on the name matching rules.\n\\li For those that take a column number, column -1 is the vector element. \n\\li For those that take a column name, I will search the vector last---if I don't find the name among the matrix columns, but the name matches the vector name, I return column -1.\n\\li If you give me both a .row and a .rowname, I go with the name; similarly for .col and\n.colname.\n\\li You can give the name of a page, e.g.\n\\code\ndouble AIC = apop_data_get(data, .rowname=\"AIC\", .col=-1, .page=\"\");\n\\endcode\n\n\\li Numeric values default to zero, which is how the examples above that treated the \\ref apop_data set as a vector or scalar could do so relatively gracefully.\nSo apop_data_get(dataset, 1) gets item (1, 0) from the matrix\nelement of \\c dataset. But as a do-what-I-mean exception, if there is no matrix element\nbut there is a vector, then this form will get vector element 1. Relying on this DWIM\nexception is useful iff you can guarantee that a data set will have only a vector or\na matrix but not both. Otherwise, be explicit and use apop_data_get(dataset, 1, -1).\n\nThe \\ref apop_data_ptr function follows the lead of \\c gsl_vector_ptr and \\c\ngsl_matrix_ptr, and like those functions, returns a pointer to the appropriate \\c double.\nFor example, to increment the (3,7)th element of an \\ref apop_data set:\n\n\\code\n(*apop_data_ptr(dataset, 3, 7))++;\n\\endcode\n\n\\li\\ref apop_data_get\n\\li\\ref apop_data_set\n\\li\\ref apop_data_ptr : returns a pointer to the element.\n\\li\\ref apop_data_get_page : retrieve a named page from a data set. If you only need a few items, you can specify a page name to \\c apop_data_(get|set|ptr).\n\n See also:\n\n\\li double gsl_matrix_get (const gsl_matrix * m, size_t i, size_t j)\n\\li double gsl_vector_get (const gsl_vector * v, size_t i)\n\\li void gsl_matrix_set (gsl_matrix * m, size_t i, size_t j, double x)\n\\li void gsl_vector_set (gsl_vector * v, size_t i, double x)\n\\li double * gsl_matrix_ptr (gsl_matrix * m, size_t i, size_t j)\n\\li double * gsl_vector_ptr (gsl_vector * v, size_t i)\n\\li const double * gsl_matrix_const_ptr (const gsl_matrix * m, size_t i, size_t j)\n\\li const double * gsl_vector_const_ptr (const gsl_vector * v, size_t i)\n\\li gsl_matrix_get_row (gsl_vector * v, const gsl_matrix * m, size_t i)\n\\li gsl_matrix_get_col (gsl_vector * v, const gsl_matrix * m, size_t j)\n\\li gsl_matrix_set_row (gsl_matrix * m, size_t i, const gsl_vector * v)\n\\li gsl_matrix_set_col (gsl_matrix * m, size_t j, const gsl_vector * v)\n\n\n\\section mapply Map/apply\n\n\\anchor outline_mapply\nThese functions allow you to send each element of a vector or matrix to a function, either producing a new matrix (map) or transforming the original (apply). The \\c ..._sum functions return the sum of the mapped output.\n\nThere are two types, which were developed at different times. The \\ref apop_map and\n\\ref apop_map_sum functions use variadic function inputs to cover a lot of different\ntypes of process depending on the inputs. Other functions with types in their names,\nlike \\ref apop_matrix_map and \\ref apop_vector_apply, may be easier to use in some\ncases. They use the same routines internally, so use whichever type is convenient.\n\nYou can do many things quickly with these functions.\n\nGet the sum of squares of a vector's elements:\n\n\\code\n //given apop_data *dataset and gsl_vector *v:\ndouble sum_of_squares = apop_map_sum(dataset, gsl_pow_2);\ndouble sum_of_sqvares = apop_vector_map_sum(v, gsl_pow_2);\n\\endcode\n\nCreate an index vector [\\f$0, 1, 2, ...\\f$].\n\n\\code\ndouble index(double in, int index){return index;}\napop_data *d = apop_map(apop_data_alloc(100), .fn_di=index, .inplace='y');\n\\endcode\n\nGiven your log likelihood function, which acts on a \\ref apop_data set with only one\nrow, and a data set where each row of the matrix is an observation, find the total\nlog likelihood via:\n\n\\code\nstatic double your_log_likelihood_fn(apop_data * in)\n {[your math goes here]}\n\ndouble total_ll = apop_map_sum(dataset, .fn_r=your_log_likelihood_fn);\n\\endcode\n\nHow many missing elements are there in your data matrix? \n\n\\code\nstatic double nan_check(const double in){ return isnan(in);}\n\nint missing_ct = apop_map_sum(in, nan_check, .part='m');\n\\endcode\n\nGet the mean of the not-NaN elements of a data set:\n\n\\code\nstatic double no_nan_val(const double in){ return isnan(in)? 0 : in;}\nstatic double not_nan_check(const double in){ return !isnan(in);}\n\nstatic double apop_mean_no_nans(apop_data *in){\n return apop_map_sum(in, no_nan_val)/apop_map_sum(in, not_nan_check);\n}\n\\endcode\n\nThe following program randomly generates a data set where each row is a list of numbers with a different mean. It then finds the \\f$t\\f$ statistic for each row, and the confidence with which we reject the claim that the statistic is less than or equal to zero.\n\nNotice how the older \\ref apop_vector_apply uses file-global variables to pass information into the functions, while the \\ref apop_map uses a pointer to send parameters to the functions.\n\n\\include t_test_by_rows.c\n\nOne more toy example demonstrating the use of \\ref apop_map and \\ref apop_map_sum :\n\n\\include apop_map_row.c\n\n\n\\li If the number of threads is greater than one, then the matrix will be broken\ninto chunks and each sent to a different thread. Notice that the GSL is generally\nthreadsafe, and SQLite is threadsafe conditional on several commonsense caveats that\nyou'll find in the SQLite documentation. See \\ref apop_rng_get_thread() to use the GSL's RNGs in a threaded environment.\n\n\\li The \\c ...sum functions are convenience functions that call \\c ...map and then add up the contents. Thus, you will need to have adequate memory for the allocation of the temp matrix/vector.\n\n\\li\\ref apop_map\n\\li\\ref apop_map_sum\n\\li\\ref apop_matrix_apply\n\\li\\ref apop_matrix_map\n\\li\\ref apop_matrix_map_all_sum\n\\li\\ref apop_matrix_map_sum\n\\li\\ref apop_vector_apply\n\\li\\ref apop_vector_map\n\\li\\ref apop_vector_map_sum\n\n\n\n\\section matrixmathtwo Basic Math\n\n\\li\\ref apop_vector_exp : exponentiate every element of a vector\n\\li\\ref apop_vector_log : take the natural log of every element of a vector\n\\li\\ref apop_vector_log10 : take the log (base 10) of every element of a vector\n\\li\\ref apop_vector_distance : find the distance between two vectors via various metrics\n\\li\\ref apop_vector_normalize : scale/shift a matrix to have mean zero, sum to one, have a range of exactly \\f$[0, 1]\\f$, et cetera\n\\li\\ref apop_vector_entropy : calculate the entropy of a vector of frequencies or probabilities\n\nSee also:\n\n\\li int gsl_matrix_add (gsl_matrix * a, const gsl_matrix * b)\n\\li int gsl_matrix_sub (gsl_matrix * a, const gsl_matrix * b)\n\\li int gsl_matrix_mul_elements (gsl_matrix * a, const gsl_matrix * b)\n\\li int gsl_matrix_div_elements (gsl_matrix * a, const gsl_matrix * b)\n\\li int gsl_matrix_scale (gsl_matrix * a, const double x)\n\\li int gsl_matrix_add_constant (gsl_matrix * a, const double x)\n\\li gsl_vector_add (gsl_vector * a, const gsl_vector * b)\n\\li gsl_vector_sub (gsl_vector * a, const gsl_vector * b)\n\\li gsl_vector_mul (gsl_vector * a, const gsl_vector * b)\n\\li gsl_vector_div (gsl_vector * a, const gsl_vector * b)\n\\li gsl_vector_scale (gsl_vector * a, const double x)\n\\li gsl_vector_add_constant (gsl_vector * a, const double x)\n\n \n\\section matrixmath Matrix math\n\n\\li\\ref apop_dot : matrix \\f$\\cdot\\f$ matrix, matrix \\f$\\cdot\\f$ vector, or vector \\f$\\cdot\\f$ matrix\n\\li\\ref apop_matrix_determinant\n\\li\\ref apop_matrix_inverse\n\\li\\ref apop_det_and_inv : find determinant and inverse at the same time\n\nSee the GSL documentation for myriad further options.\n\n\n\\section sumstats Summary stats\n\n\\li\\ref apop_data_summarize\n\\li\\ref apop_vector_moving_average\n\\li\\ref apop_vector_percentiles\n\\li\\ref apop_vector_bounded\n\nSee also:\n\n\\li double gsl_matrix_max (const gsl_matrix * m)\n\\li double gsl_matrix_min (const gsl_matrix * m)\n\\li void gsl_matrix_minmax (const gsl_matrix * m, double * min_out, double * max_out)\n\\li void gsl_matrix_max_index (const gsl_matrix * m, size_t * imax, size_t * jmax)\n\\li void gsl_matrix_min_index (const gsl_matrix * m, size_t * imin, size_t * jmin)\n\\li void gsl_matrix_minmax_index (const gsl_matrix * m, size_t * imin, size_t * jmin, size_t * imax, size_t * jmax)\n\\li gsl_vector_max (const gsl_vector * v)\n\\li gsl_vector_min (const gsl_vector * v)\n\\li gsl_vector_minmax (const gsl_vector * v, double * min_out, double * max_out)\n\\li gsl_vector_max_index (const gsl_vector * v)\n\\li gsl_vector_min_index (const gsl_vector * v)\n\\li gsl_vector_minmax_index (const gsl_vector * v, size_t * imin, size_t * imax)\n\n\n\\section moments Moments\n\nFor most of these, you can add a weights vector for weighted mean/var/cov/..., such as\napop_vector_mean(d->vector, .weights=d->weights)\n\n\\li\\ref apop_mean : the first three with short names operate on a vector.\n\\li\\ref apop_sum\n\\li\\ref apop_var\n\\li\\ref apop_matrix_sum\n\\li\\ref apop_data_correlation\n\\li\\ref apop_data_covariance\n\\li\\ref apop_data_summarize\n\\li\\ref apop_matrix_mean\n\\li\\ref apop_matrix_mean_and_var\n\\li\\ref apop_vector_correlation\n\\li\\ref apop_vector_cov\n\\li\\ref apop_vector_kurtosis\n\\li\\ref apop_vector_kurtosis_pop \n\\li\\ref apop_vector_mean\n\\li\\ref apop_vector_skew\n\\li\\ref apop_vector_skew_pop\n\\li\\ref apop_vector_sum\n\\li\\ref apop_vector_var\n\\li\\ref apop_vector_var_m \n\n\n\\section convsec Conversion among types\n\nThere are no functions provided to convert from \\ref apop_data to the constituent\nelements, because you don't need a function.\n\nIf you need an individual element, you can use its pointer to retrieve it:\n\n\\code\napop_data *d = apop_query_to_mixed_data(\"vmmw\", \"select result, age, \"\n \"income, replicate_weight from data\");\ndouble avg_result = apop_vector_mean(d->vector, .weights=d->weights);\n\\endcode\n\nIn the other direction, you can use compound literals to wrap an \\ref apop_data struct\naround a loose vector or matrix:\n\n\\code\n//Given:\ngsl_vector *v;\ngsl_matrix *m;\n\n// Then this form wraps the elements into automatically-allocated apop_data structs.\n\napop_data *dv = &(apop_data){.vector=v}; \napop_data *dm = &(apop_data){.matrix=m};\n\napop_data *v_dot_m = apop_dot(dv, dm);\n\n//Here is a macro to hide C's ugliness:\n#define As_data(...) (&(apop_data){__VA_ARGS__})\n\napop_data *v_dot_m2 = apop_dot(As_data(.vector=v), As_data(.matrix=m));\n\n//The wrapped object is an automatically-allocated structure pointing to the\n//original data. If it needs to persist or be separate from the original,\n//make a copy:\napop_data *dm_copy = apop_data_copy(As_data(.vector=v, .matrix=m));\n\\endcode\n\n\\li\\ref apop_array_to_vector : double*\\f$\\to\\f$ gsl_vector\n\\li\\ref apop_data_fill : double*\\f$\\to\\f$ \\ref apop_data\n\\li\\ref apop_data_falloc : macro to allocate and fill a \\ref apop_data set\n\\li\\ref apop_text_to_data : delimited text file\\f$\\to\\f$ \\ref apop_data\n\\li\\ref apop_text_to_db : delimited text file\\f$\\to\\f$ database table\n\\li\\ref apop_vector_to_matrix\n\n\n\\section names Name handling\n\nIf you generate your data set via \\ref apop_text_to_data or from the database via\n\\ref apop_query_to_data (or \\ref apop_query_to_text or \\ref apop_query_to_mixed_data)\nthen column names appear as expected. Set apop_opts.db_name_column to the\nname of a column in your query result to use that column name for row names.\n\nSample uses, given \\ref apop_data set d:\n\n\\code\nint row_name_count = d->names->rowct\nint col_name_count = d->names->colct\nint text_name_count = d->names->textct\n\n//Manually add names in sequence:\napop_name_add(d->names, \"the vector\", 'v');\napop_name_add(d->names, \"row 0\", 'r');\napop_name_add(d->names, \"row 1\", 'r');\napop_name_add(d->names, \"row 2\", 'r');\napop_name_add(d->names, \"numeric column 0\", 'c');\napop_name_add(d->names, \"text column 0\", 't');\napop_name_add(d->names, \"The name of the data set.\", 'h');\n\n//or append several names at once\napop_data_add_names(d, 'c', \"numeric column 1\", \"numeric column 2\", \"numeric column 3\");\n\n//point to element i from the row/col/text names:\n\nchar *rowname_i = d->names->row[i];\nchar *colname_i = d->names->col[i];\nchar *textname_i = d->names->text[i];\n\n//The vector also has a name:\nchar *vname = d->names->vector;\n\\endcode\n\n\\li\\ref apop_name_add : add one name\n\\li\\ref apop_data_add_names : add a sequence of names at once\n\\li\\ref apop_name_stack : copy the contents of one name list to another\n\\li\\ref apop_name_find : find the row/col number for a given name.\n\\li\\ref apop_name_print : print the \\ref apop_name struct, for diagnostic purposes.\n\n\n\\section textsec Text data\n\nThe \\ref apop_data set includes a grid of strings, named text, for holding text data. \n\nText should be encoded in UTF-8. ASCII is a subset of UTF-8, so that's OK too.\n\nThere are a few simple forms for handling the \\c text element of an \\c apop_data set.\n\n\\li Use \\ref apop_text_alloc to allocate the block of text. It is actually a realloc function, which you can use to resize an existing block without leaks. See the example below.\n\\li Use \\ref apop_text_set to write text elements. It replaces any existing text in the given slot without memory leaks.\n\\li The number of rows of text data in tdata is\ntdata->textsize[0]; \nthe number of columns is tdata->textsize[1].\n\\li Refer to individual elements using the usual 2-D array notation, tdata->text[row][col].\n\\li x[0] can always be written as *x, which may save some typing. The number of rows is *tdata->textsize. If you have a single column of text data (i.e., all data is in column zero), then item \\c i is *tdata->text[i]. If you know you have exactly one cell of text, then its value is **tdata->text.\n\\li After \\ref apop_text_alloc, all elements are the empty string \"\", which\nyou can check via \n\\code\nif (!strlen(dataset->text[i][j])) printf(\"\")\n//or\nif (!*dataset->text[i][j]) printf(\"\")\n\\endcode\nFor the sake of efficiency when dealing with large, sparse data sets, all blank cells\npoint to the same static empty string, meaning that freeing cells must be\ndone with care. Your best bet is to rely on \\ref apop_text_set, \\ref apop_text_alloc,\nand \\ref apop_text_free to do the memory management for you.\n\nHere is a sample program that uses these forms, plus a few text-handling functions.\n\n\\include eg/text_demo.c\n\n\\li\\ref apop_data_transpose() : also transposes the text data. Say that you use\ndataset = apop_query_to_text(\"select onecolumn from data\"); then you have a\nsequence of strings, d->text[0][0], d->text[1][0], .... After apop_data\n*dt = apop_data_transpose(dataset), you will have a single list of strings,\ndt->text[0], which is often useful as input to list-of-strings handling\nfunctions.\n\n\\li\\ref apop_query_to_text\n\\li\\ref apop_text_alloc : allocate or resize the text part of an \\ref apop_data set.\n\\li\\ref apop_text_set : replace a single cell of the text grid with new text.\n\\li\\ref apop_text_paste : convert a table of strings into one long string.\n\\li\\ref apop_text_unique_elements : get a sorted list of unique elements for one column of text.\n\\li\\ref apop_text_free : you may never need this, because \\ref apop_data_free calls it.\n\\li\\ref apop_regex : friendlier front-end for POSIX-standard regular expression\n searching; pulls matches into an \\ref apop_data set.\n\\li\\ref apop_text_unique_elements\n\n\\subsection fact Generating factors\n\n\\em Factor is jargon for a numbered category. Number-crunching programs prefer integers over text, so we need a function to produce a one-to-one mapping from text categories into numeric factors. \n\nA \\em dummy is a variable that is either one or zero, depending on membership in a given\ngroup. Some methods (typically when the variable is an input or independent variable\nin a regression) prefer dummies; some methods (typically for outcome or dependent\nvariables) prefer factors. The functions that generate factors and dummies will add\nan informational page to your \\ref apop_data set with a name like \\ listing the conversion from the artificial numeric factor to\nthe original data. Use \\ref apop_data_get_factor_names to get a pointer to that page.\n\nYou can use the factor table to translate from numeric categories back to text (though\nyou probably have the original text column in your data anyway).\n\nHaving the factor list in an auxiliary table makes it easy to ensure that multiple\n\\ref apop_data sets use the same single categorization scheme. Generate factors in the\nfirst set, then copy the factor list to the second, then run \\ref apop_data_to_factors\non the second:\n\n\\code\napop_data_to_factors(d1);\nd2->more = apop_data_copy(apop_data_get_factor_names(d1));\napop_data_to_factors(d2);\n\\endcode\n\nSee the documentation for \\ref apop_logit for a sample linear model using a factor dependent variable and dummy independent variable.\n\n\\li\\ref apop_data_to_dummies\n\\li\\ref apop_data_to_factors\n\\li\\ref apop_data_get_factor_names\n*/\n\n/** \\page dbs Databases\n\nThese are convenience functions to handle interaction with SQLite or mySQL/mariaDB. They open one and only one database, and handle most of the interaction therewith for you.\n\nYou will probably first use \\ref apop_text_to_db to pull data into the database, then \\ref apop_query to clean the data in the database, and finally \\ref apop_query_to_data to pull some subset of the data out for analysis.\n\n\\li In all cases, your query may be in printf form. For example:\n\\code\nchar tabname[] = \"demographics\";\nchar colname[] = \"heights\";\nint min_height = 175;\napop_query(\"select %s from %s where %s > %i\", colname, tabname, colname, min_height);\n\\endcode\n\n\nSee the \\ref db_moments section below for not-SQL-standard math functions that you can\nuse when sending queries from Apophenia, such as \\c pow, \\c stddev, or \\c sqrt.\n\n\\li \\ref apop_text_to_db : Read a text file on disk into the database. Data analysis projects often start with a call to this.\n\\li \\ref apop_data_print : If you include the argument .output_type='d', this prints your \\ref apop_data set to the database.\n\\li \\ref apop_query : Manipulate the database, return nothing (e.g., insert rows or create table).\n\\li \\ref apop_db_open : Optional, for when you want to use a database on disk.\n\\li \\ref apop_db_close : A useful (and in some cases, optional) companion to \\ref apop_db_open.\n\\li \\ref apop_table_exists : Check to make sure you aren't reinventing or destroying data. Also, a clean way to drop a table.\n\n\\li Apophenia reserves the right to insert temp tables into the opened database. They\nwill all have names beginning with apop_, so the reader is advised to not\ngenerate tables with such names, and is free to ignore or delete any such tables that\nturn up.\n\\li If you need to deal with two databases, use SQL's attach database. By default\nwith SQLite, Apophenia opens an in-memory database handle. It is a sensible workflow to\nuse the faster in-memory database as the primary database, and then attach an on-disk database\nto read in data and write final output tables.\n\n\\section edftd Extracting data from the database\n\n\\li\\ref apop_db_to_crosstab : take up to three columns in the database (row, column, value) and produce a table of values.\n\\li\\ref apop_query_to_data\n\\li\\ref apop_query_to_float\n\\li\\ref apop_query_to_mixed_data\n\\li\\ref apop_query_to_text\n\\li\\ref apop_query_to_vector\n\n\\section wdttd Writing data to the database\n\nSee the print functions at \\ref Legi. E.g.\n\n\\code\napop_data_print(yourdata, .output_type='d', .output_name=\"dbtab\");\n\\endcode\n\n\\section cmdline Command-line utilities\n\nA few functions have proven to be useful enough to be worth breaking out into their own programs, for use in scripts or other data analysis from the command line:\n\n\\li The \\c apop_text_to_db command line utility is a wrapper for the \\ref apop_text_to_db command.\n\\li The \\c apop_db_to_crosstab function is a wrapper for the \\ref apop_db_to_crosstab function.\n\n\\section db_moments Database moments (plus pow()!)\n\nSQLite lets users define new functions for use in queries, and Apophenia uses this facility to define a few common functions.\n\n\\li select ran() from table will produce a new random number between zero and one for every row of the input table, using \\c gsl_rng_uniform. \n\n\\li The SQL standard includes the count(x) and avg(x) aggregators,\nbut statisticians are usually interested in higher moments as well---at least the\nvariance. Therefore, SQL queries using the Apophenia library may include any of these moments:\n\n\\code\nselect count(x), stddev(x), avg(x), var(x), variance(x), skew(x), kurt(x), kurtosis(x),\nstd(x), stddev_samp(x), stddev_pop(x), var_samp(x), var_pop(x)\nfrom table\ngroup by whatever\n\\endcode\n\nvar and variance; kurt and kurtosis do the same thing; choose the one that sounds better to you.\nKurtosis is the fourth central moment by itself, not adjusted by subtracting three or dividing by variance squared.\nvar, var_samp, stddev and stddev_samp give sample variance/standard deviation; variance, var_pop, std and stddev_pop give population standard deviation. The plethora of variants are for mySQL compatibility.\n\n\\li The var/skew/kurtosis functions calculate sample moments. If you want the second\npopulation moment, multiply the variance by \\f$(n-1)/n\\f$; for the third population moment,\nmultiply the skew by \\f$(n-1)(n-2)/n^2\\f$. The equation for the unbiased sample kurtosis\nas calculated in Appendix M of\nModeling with Data is not quite as easy to adjust.\n\n\\li Also provided: wrapper functions for standard math library\nfunctions---sqrt(x), pow(x,y), exp(x), log(x),\nand trig functions. They call the standard math library function of the same name\nto calculate \\f$\\sqrt{x}\\f$, \\f$x^y\\f$, \\f$e^x\\f$, \\f$\\ln(x)\\f$, \\f$\\sin(x)\\f$,\n\\f$\\arcsin(x)\\f$, et cetera. For example:\n\n\\code\nselect sqrt(x), pow(x,0.5), exp(x), log(x), log10(x),\n sin(x), cos(x), tan(x), asin(x), acos(x), atan(x)\nfrom table\n\\endcode\n\n\\li The ran() function calls gsl_rng_uniform to produce a uniform\ndraw between zero and one. It uses the stock of RNGs from \\ref apop_rng_get_thread.\n\nHere is a test script using many of the above.\n\n\\include db_fns.c\n*/\n\n\n/** \\page modelsec Models\nSee \\ref gentle_model for an overview of the intent and basic use of the \\ref apop_model struct.\n\nThis segment goes into greater detail on the use of existing \\ref apop_model objects.\nIf you need to write a new model, see \\ref modeldetails.\n\nThe \\c estimate function will estimate the parameters of your model. Just prep the data, select a model, and produce an estimate:\n\n\\code\n apop_data *data = apop_query_to_data(\"select outcome, in1, in2, in3 from dataset\");\n apop_model *the_estimate = apop_estimate(data, apop_probit);\n apop_model_print(the_estimate);\n\\endcode\n\nAlong the way to estimating the parameters, most models also find covariance estimates for\nthe parameters, calculate statistics like log likelihood, and so on, which the final print statement will show.\n\nThe apop_probit model that ships with Apophenia is unparameterized:\napop_probit->parameters==NULL. The output from the estimation,\nthe_estimate, has the same form as apop_probit, but\nthe_estimate->parameters has a meaningful value.\n\nApophenia ships with many well-known models for your immediate use, including\nprobability distributions, such as the \\ref apop_normal, \\ref apop_poisson, or \\ref\napop_beta models. The data is assumed to have been drawn from a given distribution and\nthe question is only what distributional parameters best fit. For example, given that\nthe data is Normally distributed, find \\f$\\mu\\f$ and \\f$\\sigma\\f$ via\napop_estimate(your_data, apop_normal).\n\nThere are also linear models like \\ref apop_ols, \\ref apop_probit, and \\ref apop_logit. As in the example, they are on equal footing with the distributions, so nothing keeps you from making random draws from an estimated linear model.\n\n \\li If you send a data set with the \\c weights vector filled, \\ref apop_ols estimates Weighted OLS.\n \\li If the dependent variable has more than two categories, the \\ref apop_probit and\n\\ref apop_logit models estimate a multinomial logit or probit.\n \\li There are separate \\ref apop_normal and \\ref apop_multivariate_normal functions\nbecause the parameter formats are slightly different: the univariate Normal keeps both\n\\f$\\mu\\f$ and \\f$\\sigma\\f$ in the vector element of the parameters; the multivariate\nversion uses the vector for the vector of means and the matrix for the \\f$\\Sigma\\f$\nmatrix. The univariate version is so heavily used that it merits a special-case model.\n\nSee the \\ref models page for a list of models shipped with Apophenia,\nincluding popular favorites like \\ref apop_beta, \\ref apop_binomial, \\ref apop_iv\n(instrumental variables), \\ref apop_kernel_density, \\ref apop_loess, \\ref apop_lognormal,\n\\ref apop_pmf (see \\ref histosec below), and \\ref apop_poisson.\n\nSimulation models seem to not fit this form, but you will see below that if you can write an objective function for the \\c p method of the model, you can use the above tools. Notably, you can estimate parameters via maximum likelihood and then give confidence intervals around those parameters.\n\nMore estimation output\n\nIn the \\ref apop_model returned by \\ref apop_estimate, you will find:\n\n \\li The actual parameter estimates are in an \\ref apop_data set at \\c your_model->parameters.\n \\li A pointer to the \\ref apop_data set used for estimation, named \\c data.\n \\li Scalar statistics of the model listed in the output model's \\c info group,\nwhich may include some hypothesis tests, a list of expected values, log likelihood, AIC, AIC_c, BIC, et cetera.\nThese can be retrieved via a form like\n\\code\napop_data_get(your_model->info, .rowname=\"log likelihood\");\n//or\napop_data_get(your_model->info, .rowname=\"AIC\");\n\\endcode\nIf those are not necessary, adding to your model an \\ref apop_parts_wanted_settings\ngroup with its default values (see below on settings groups) signals to the model\nthat you want only the parameters and to not waste possibly significant CPU time\non covariances, expected values, et cetera. See the \\ref apop_parts_wanted_settings\ndocumentation for examples and further refinements.\n \\li In many cases, covariances of the parameters as a page appended to the parameters; retrieve via\n\\code\napop_data *cov = apop_data_get_page(your_model->parameters, \"\");\n\\endcode\n \\li Typically for regression-type models, the table of expected values (typically including\nexpected value, actual value, and residual) is a page stapled to the main info\npage. Retrieve via:\n\\code\napop_data *predict = apop_data_get_page(your_model->info, \"\");\n\\endcode\n\nSee individual model documentation for what is provided by any given model.\n\nPost-estimation uses\n\nBut we expect much more from a model than estimating parameters from data. \n\nContinuing the above example where we got an estimated Probit model named \\c the_estimate, we can interrogate the estimate in various familiar ways:\n\n\\code\napop_data *expected_value = apop_predict(NULL, the_estimate);\n\ndouble density_under = apop_cdf(expected_value, the_estimate);\n\napop_data *draws = apop_model_draws(the_estimate, .count=1000);\n\\endcode\n\n\\subpage dataones\n\n\\section modelparameterization Parameterizing or initializing a model\n\nThe models that ship with Apophenia have the requisite procedures for estimation,\nmaking draws, and so on, but have parameters==NULL and settings==NULL. The\nmodel is thus, for many purposes, incomplete, and you will need to take some action to\ncomplete the model. As per the examples to follow, there are several possibilities:\n\n \\li Estimate it! Almost all models can be sent with a data set as an argument to the\napop_estimate function. The input model is unchanged, but the output model\nhas parameters and settings in place.\n \\li If your model has a fixed number of numeric parameters, then you can set them with\n\\ref apop_model_set_parameters.\n \\li If your model has a variable number of parameters, you can directly set the \n\\c parameters element via \\ref apop_data_falloc. For most purposes, you will also need to\nset the \\c msize1, \\c msize2, \\c vsize, and \\c dsize elements to the size you want. See\nthe example below.\n \\li Some models have disparate, non-numeric settings rather than a simple matrix of\nparameters. For example, an kernel density estimate needs a model as a kernel and a\nbase data set, which can be set via \\ref apop_model_set_settings.\n\nHere is an example that shows the options for parameterizing a model. After each\nparameterization, 20 draws are made and written to a file named draws-[modelname].\n\n\\include ../eg/parameterization.c\n\n\n\\section transformsec Filtering & updating\n\nThe model structure makes it easy to generate new models that are variants of prior\nmodels. Bayesian updating, for example, takes in one \\ref apop_model that we call the\nprior, one \\ref apop_model that we call a likelihood, and outputs an \\ref apop_model\nthat we call the posterior. One can produce complex models using simpler transformations\nas well. For example, \\ref apop_model_fix_params will set the free parameters of\nan input model to a fixed value, thus producing a model with fewer parameters. To\ntransform a Normal(\\f$\\mu\\f$, \\f$\\sigma\\f$) into a one-parameter Normal(\\f$\\mu\\f$, 1):\n\n\\code\napop_model *N_sigma1 = apop_model_fix_params(apop_model_set_parameters(apop_normal, NAN, 1));\n\\endcode\n\nThis can be used anywhere the original Normal distribution can be. To give another\nexample, if we need to truncate the distribution in the data space:\n\n\\code\n//The constraint function.\ndouble over_zero(apop_data *in, apop_model *m){\n return apop_data_get(in) > 0;\n}\n\napop_model *trunc = apop_model_dconstrain(.base_model=N_sigma1,\n .constraint=over_zero);\n\\endcode\n\nChaining together simpler transformations is an easy method to produce \nmodels of arbitrary detail. In the following example:\n\n\\li Nature generated data using a mixture of three Poisson distributions,\nwith \\f$\\lambda=2.8\\f$, \\f$2.0\\f$, and \\f$1.3\\f$.\nThe resulting model is generated using \\ref apop_model_mixture. \n\\li Not knowing the true distribution, the analyst \nmodels the data with a single Poisson\\f$(\\lambda)\\f$ distribution with a prior on \\f$\\lambda\\f$.\nThe prior selected is a\ntruncated Normal(2, 1), generated by sending the stock\n\\ref apop_normal model to the data-space constraint function \\ref apop_dconstrain.\n\\li The \\ref apop_update function takes three arguments: the data set, which comes from\ndraws from the mixture, the prior, and the likelihood. It produces an output model\nwhich, in this case, is a PMF describing a distribution over \\f$\\lambda\\f$, because\na truncated Normal and a Poisson are not conjugate distributions. Knowing that it is\na PMF, the ->data element holds a set of draws from the posterior.\n\\li The analyst would like to present an approximation to the posterior in a simpler form,\nand so finds the parameters \\f$\\mu\\f$ and \\f$\\sigma\\f$ of the Normal distribution that\nis closest to that posterior.\n\nHere is a program---almost a single line of code---that builds the final approximation to the posterior\nmodel from the subcomponents, including draws from Nature and the analyst's prior\nand likelihood:\n\n\\include ../eg/transform.c\n\n\\section mathmethods Model methods\n\n\\li\\ref apop_estimate : estimate the parameters of the model with data.\n\\li\\ref apop_predict : the expected value function.\n\\li\\ref apop_draw : random draws from an estimated model.\n\\li\\ref apop_p : the probability of a given data set given the model.\n\\li\\ref apop_log_likelihood : the log of \\ref apop_p\n\\li\\ref apop_score : the derivative of \\ref apop_log_likelihood\n\\li\\ref apop_model_print : write model components to the screen or a file\n\\li\\ref apop_model_copy : duplicate a model\n\\li\\ref apop_model_set_parameters : Use this to convert a Normal(\\f$\\mu\\f$, \\f$\\sigma\\f$) with unknown \\f$\\mu\\f$ and \\f$\\sigma\\f$ into a Normal(0, 1), for example.\n\\li\\ref apop_model_free\n\\li\\ref apop_model_clear , \\ref apop_prep : remove the parameters from a parameterized model. Used infrequently.\n\\li\\ref apop_model_draws : many random draws from an estimated model.\n\n\n\n\\li\\ref apop_update : Bayesian updating\n\\li\\ref apop_model_coordinate_transform : apply an invertible transformation to the data space\n\\li\\ref apop_model_dconstrain : constrain the data space of a model to a subspace. E.g., truncate a Normal distribution so \\f$x>0\\f$.\n\\li\\ref apop_model_fix_params : hold some parameters constant\n\\li\\ref apop_model_mixture : a linear combination of models\n\\li\\ref apop_model_cross : If \\f$(d_1)\\f$ has a Normal\\f$(\\mu, \\sigma)\\f$ distribution\nand \\f$d_2\\f$ has an independent Poisson\\f$(\\lambda)\\f$ distribution, then \\f$(d_1,\nd_2)\\f$ has an apop_model_cross(apop_normal, apop_poisson) distribution with\nparameters \\f$(\\mu, \\sigma, \\lambda)\\f$.\n\n\n\\section modelsettings Settings groups\n\nDescribing a statistical, agent-based, social, or physical model in a standardized\nform is difficult because every model has significantly different settings. An\nMLE requires a method of search (conjugate gradient, simplex, simulated annealing),\nand a histogram needs the number of bins to be filled with data.\n\nSo, the \\ref apop_model includes a single list which can hold an arbitrary number of settings groups, like the search specifications for finding the maximum likelihood, a histogram for making random draws, and options about the model type.\n\nSettings groups are automatically initialized with default values when\nneeded. If the defaults do no harm, then you don't need to think about\nthese settings groups at all.\n\nHere is an example where a settings group is worth tweaking: the \\ref apop_parts_wanted_settings group indicates which parts\nof the auxiliary data you want. \n\n\\code\n1 apop_model *m = apop_model_copy(apop_ols);\n2 Apop_settings_add_group(m, apop_parts_wanted, .covariance='y');\n3 apop_model *est = apop_estimate(data, m);\n\\endcode\n\n\nLine one establishes the baseline form of the model. Line two adds a settings group\nof type \\ref apop_parts_wanted_settings to the model. By default other auxiliary items, like the expected values, are set to \\c 'n' when using this group, so this specifies that we want covariance and only covariance. Having stated our preferences, line three does the estimation we want.\n\nNotice that the \\c _settings ending to the settings group's name isn't written---macros\nmake it happen. The remaining arguments to \\c Apop_settings_add_group (if any) follow\nthe \\ref designated syntax of the form .setting=value.\n\nThere is an \\ref apop_model_copy_set macro that adds a settings group when it is first copied, joining up lines one and two above:\n\n\\code\napop_model *m = apop_model_copy_set(apop_ols, apop_parts_wanted, .covariance='y');\n\\endcode\n\nSettings groups are copied with the model, which facilitates chaining\nestimations. Continuing the above example, you could re-estimate to get the predicted\nvalues and covariance via:\n\n\\code\nApop_settings_set(est, apop_parts_wanted, predicted, 'y');\napop_model *est2 = apop_estimate(data, est);\n\\endcode\n\nMaximum likelihood search has many settings that could be modified, and so provides\nanother common example of using settings groups:\n\n\\code\napop_model *the_estimate = apop_estimate(data, apop_probit);\n\n//Redo the Probit's MLE search using Newton's Method:\nApop_settings_add_group(the_estimate, apop_mle, .verbose='y', \n .tolerance=1e-4, .method=\"Newton\");\napop_model *re_est = apop_estimate(data, the_estimate);\n\\endcode\n\nTo clarify the distinction between parameters and settings, note that parameters are\nestimated from the data, often via a maximum likelihood search. In an ML search,\nthe method of search, the number of bins in a histogram, or the number of steps in a\nsimulation would be held fixed as the search iterates over possible parameters (and\nif these settings do change, then that is a meta-model that could be encapsulated into another\n\\ref apop_model). As a consequence, parameters are always numeric, while settings may\nbe any type.\n\n\\li \\ref Apop_settings_set, for modifying a single setting, doesn't use the designated initializers format.\n\\li Because the settings groups are buried within the model, debugging them can be a\npain. Here is a documented macro for \\c gdb that will help you pull a settings group out of a \nmodel for your inspection, to cut and paste into your .gdbinit. It shouldn't be too difficult to modify this macro for other debuggers.\n\n\\code\ndefine get_group\n set $group = ($arg1_settings *) apop_settings_get_grp( $arg0, \"$arg1\", 0 )\n p *$group\nend\ndocument get_group \nGets a settings group from a model.\nGive the model name and the name of the group, like\nget_group my_model apop_mle \nand I will set a gdb variable named $group that points to that model, \nwhich you can use like any other pointer. For example, print the contents with\np *$group\nThe contents of $group are printed to the screen as visible output to this macro.\nend \n\\endcode\n\nFor using a model, that's all of what you need to know. For details on writing a new settings group, see \\ref settingswriting .\n\n\\li\\ref Apop_settings_add_group\n\\li\\ref Apop_settings_set\n\\li\\ref Apop_settings_get : get a single element from a settings group.\n\\li\\ref Apop_settings_get_group : get the whole settings group.\n*/\n\n/** \\page dataones Data format for regression-type models\n\nRegression-type estimations typically require a constant column. That is, the 0th\ncolumn of the data is a constant (one), so the parameter \\f$\\beta_0\\f$ is\nslightly special in corresponding to a constant rather than a variable.\n\nSome stats packages implicitly assume a constant column, which the user never\nsees. This violates the principle of transparency upon which Apophenia is based.\nGiven a data matrix \\f$X\\f$ with the estimated parameters\n\\f$\\beta\\f$, if the model asserts that the product \\f$X\\beta\\f$ has meaning, then you\nshould be able to easily calculate that product. With a ones column, a dot product is one line:\napop_dot(x, your_est->parameters); without a ones column, one would basically\nhave to construct one (using \\c gsl_matrix_set_all and \\c apop_data_stack).\n\nEach regression-type estimation has one dependent variable and several independent. In\nthe end, we want the dependent variable to be in the vector element. Removing\na column from a gsl_matrix and adjusting all subsequent columns is relatively\ndifficult, because (like most structs built with the aim of very efficient processing) the\nstruct depends on an equal spacing in memory between each element.\n\n The automatic case\n\nWe can resolve both the need for a ones column and for having the dependent column in\nthe vector at the same time. Given a data set with no vector element and the dependent\nvariable in the first column of the matrix, we can copy the dependent variable into\nthe vector and then replace the first column of the matrix with ones. The result fits\nall of the above expectations.\n\nYou as a user merely have to send in a \\c apop_data set with \\c NULL vector and a dependent\ncolumn in the first column. If the data is coming from the database, then the query\nis natural:\n\n\\code\napop_data *regression_data = apop_query_to_data(\"select depvar, indyvar1, indyvar2, indyvar3 from dataset\");\napop_model_print(apop_estimate(regression_data, apop_ols));\n\\endcode\n\n The already-prepped case\n\nIf your data has a vector element, then the prep routines won't change anything.\nIf you don't want to use a constant column, or your data has already been prepped by\nan estimation, then this is what you want.\n\n\\code\napop_data *regression_data = apop_query_to_mixed_data(\"vmmm\", \"select depvar, indyvar1, indvar2, indvar3 from dataset\");\napop_model_print(apop_estimate(regression_data, apop_logit));\n\\endcode\n*/\n\n\n/** \\page testpage Tests & diagnostics\n\nHere is the model for all hypothesis testing within Apophenia:\n\n\\li Calculate a statistic.\n\\li Describe the distribution of that statistic.\n\\li Work out how much of the distribution is (above|below|closer to zero than) the statistic.\n\nThere are a handful of named tests that produce a known statistic and then compare to a\nknown distribution, like \\ref apop_test_kolmogorov or \\ref apop_test_fisher_exact. For\ntraditional distributions (Normal, \\f$t\\f$, \\f$\\chi^2\\f$), use the \\ref apop_test convenience\nfunction.\n\nIn especially common cases, like the parameters from an OLS regression,\nthe commonly-associated \\f$t\\f$ test is included as part of the estimation\noutput, typically as a row in the \\c info element of the output \\ref apop_model.\n\n\n\\li\\ref apop_test\n\\li\\ref apop_paired_t_test\n\\li\\ref apop_f_test\n\\li\\ref apop_t_test\n\\li\\ref apop_test_anova_independence\n\\li\\ref apop_test_fisher_exact\n\\li\\ref apop_test_kolmogorov\n\\li\\ref apop_estimate_coefficient_of_determination\n\\li\\ref apop_estimate_r_squared\n\nSee also these Monte Carlo methods:\n\n\\li\\ref apop_bootstrap_cov\n\\li\\ref apop_jackknife_cov\n\nTo give another example of testing, here is a function that was briefly a part of\nApophenia, but seemed a bit out of place. Here it is as a sample:\n\n\\code\n// Input: any vector, which will be normalized in place. Output: 1 - the p-value\n// for a chi-squared test to answer the question, \"with what confidence can I\n// reject the hypothesis that the variance of my data is zero?\"\n\ndouble apop_test_chi_squared_var_not_zero(gsl_vector *in){\n Apop_stopif(!in, return NAN, 0, \"input vector is NULL. Doing nothing.\");\n apop_vector_normalize(in, .normalization_type='s');\n double sum=apop_vector_map_sum(in, gsl_pow_2);\n return gsl_cdf_chisq_P(sum, in->size);\n}\n\\endcode\n\nOr, consider the Rao statistic, \n\\f${\\partial\\over \\partial\\beta}\\log L(\\beta)'I^{-1}(\\beta){\\partial\\over \\partial\\beta}\\log L(\\beta)\\f$\nwhere \\f$L\\f$ is a model's likelihood function and \\f$I\\f$ its information matrix. In code:\n\n\\code\napop_data * infoinv = apop_model_numerical_covariance(data, your_model);\napop_data * score = &(apop_data*){.vector=apop_numerical_gradient(data, your_model)};\napop_data * stat = apop_dot(apop_dot(score, infoinv), score);\n\\endcode\n\nGiven the correct assumptions, this is \\f$\\sim \\chi^2_m\\f$, where \\f$m\\f$ is the dimension of \\f$\\beta\\f$, so the odds of a Type I error given the model is:\n\n\\code\ndouble p_value = apop_test(stat, \"chi squared\", beta->size);\n\\endcode\n\nGeneralized parameter tests\n\nBut if your model is not from the textbook, then you have the tools to apply the\nabove three-step process to the parameters of any \\ref apop_model.\n\n\\li Model parameters are a statistic, and you know that apop_estimate(your_data,\n your_model) will output a model with a parameters element.\n\\li \\ref apop_parameter_model will return an \\ref apop_model describing \n the distribution of these parameters.\n\\li We now have the two ingredients to send to \\ref apop_cdf, which takes in a model\nand a data point and returns the area under the data point.\n\nDefaults for the parameter models are filled in via bootstrapping or resampling, meaning\nthat if your model's parameters are decidedly off the Normal path, you can still test\nclaims about the parameters.\n\nThe introductory example in \\ref gentle ran a standard OLS regression, whose output includes some\nstandard hypothesis tests; to conclude, let us go the long way and replicate those results\nvia the general \\ref apop_parameter_model mechanism. The results here will of course be\nidentical, but the more general mechanism can be used in situations where the standard\nmodels don't apply.\n\nThe first part of this program is identical to the introductory program, using \\c\nss08pdc.csv if you have downloaded it as per the instructions in \\ref gentle, or a\nsimple sample data set if not. The second half executes the three steps uses many\nof the above features: one of the inputs to \\ref apop_parameter_model (which row of\nthe parameter set to use) is sent by adding a settings group, we pull that row into\na separate data set using \\ref Apop_r, and we set its vector value by referring to it\nas the -1st element.\n\n\\include ols2.c\n\nNote that the procedure did not assume the model parameters had a certain form. It\nqueried the model for the distribution of parameter \\c agep, and if the model didn't have\na closed-form answer then a distribution via bootstrap would be provided. Then that model\nwas queried for its CDF. [The procedure does assume a symmetric distribution. Fixing this\nis left as an exercise for the reader.] For a model like OLS, this is entirely overkill, \nwhich is why OLS provides the basic hypothesis tests automatically. But for models\nwhere the distribution of parameters is unknown or has no closed-form solution, this\nmay be the only recourse.\n*/\n\n/** \\page histosec Empirical distributions and PMFs (probability mass functions)\n\nThe \\ref apop_pmf model wraps an \\ref apop_data set so it can be read as an empirical\nmodel, with a likelihood function (equal to the associated weight for observed\nvalues and zero for unobserved values), a random number generator (which \nsimply makes weighted random draws from the data), and so on. Setting it up is a\nmodel estimation from data like any other, done via \\ref apop_estimate(\\c your_data,\n\\ref apop_pmf).\n\nYou have the option of cleaning up the data before turning it into a PMF. For example...\n\n\\code\napop_data_pmf_compress(your_data); //remove duplicates\napop_data_sort(your_data);\napop_vector_normalize(your_data->weights); //weights sum to one\napop_model *a_pmf = apop_estimate(your_data, apop_pmf);\n\\endcode\n\nThese are largely optional.\n\n\\li The CDF is calculated based on the percent of the weights between the zeroth row of the PMF\nand the row specified. This generally makes more sense after \\ref apop_data_sort.\n\\li Compression produces a corresponding improvement in efficiency when first calculating\nCDFs, but is otherwise not necessary.\n\\li Sorting or normalizing is not necessary for making draws or getting a likelihood or log likelihood.\n\nIt is the \\c weights vector that holds the density represented by each row; the rest of the row represents the coordinates of that density. If the input data set has no \\c weights segment, then I assume that all rows have equal weight.\n\nFor a PMF model, the \\c parameters are \\c NULL, and the \\c data itself\nis used for calculation. Therefore, modifying the data post-estimation can break some\ninternal settings set during estimation. If you modify the data, throw away any existing\nPMFs (via \\ref apop_model_free) and re-estimate a new one.\n\n\\section histocompare Comparing histograms\n\nUsing \\ref apop_data_pmf_compress puts the data into one bin for each unique value in\nthe data set. You may instead want bins of fixed with, in the style of a histogram,\nwhich you can get via \\ref apop_data_to_bins. It requires a bin specification. If you\nsend a \\c NULL binspec, then the offset is zero and the bin size is big enough to ensure\nthat there are \\f$\\sqrt{N}\\f$ bins from minimum to maximum. The binspec will be added\nas a page to the data set, named \"\". See the \\ref apop_data_to_bins\ndocumentation on how to write a custom bin spec.\n\n\nThere are a few ways of testing the claim that one distribution equals another, typically an empirical PMF versus a smooth theoretical distribution. In both cases, you will need two distributions based on the same binspec. \n\nFor example, if you do not have a prior binspec in mind, then you can use the one generated by the first call to the histogram binning function to make sure that the second data set is in sync:\n\n\\code\napop_data_to_bins(first_set, NULL);\napop_data_to_bins(second_set, apop_data_get_page(first_set, \"\"));\n\\endcode\n\nYou can use \\ref apop_test_kolmogorov or \\ref apop_histograms_test_goodness_of_fit to generate the appropriate statistics from the pairs of bins.\n\nKernel density estimation will produce a smoothed PDF. See \\ref apop_kernel_density for details.\nOr, use \\ref apop_vector_moving_average for a simpler smoothing method.\n\n\n\\li\\ref apop_data_pmf_compress() : merge together redundant rows in a data set before calling \n \\ref apop_estimate(\\c your_data, \\ref apop_pmf); optional.\n\\li\\ref apop_vector_moving_average() : smooth a vector (e.g., your_pmf->data->weights) via moving average.\n\\li\\ref apop_histograms_test_goodness_of_fit() : goodness-of-fit via \\f$\\chi^2\\f$ statistic\n\\li\\ref apop_test_kolmogorov() : goodness-of-fit via Kolmogorov-Smirnov statistic\n\\li\\ref apop_kl_divergence() : measure the information loss from one (typically empirical) distribution to another distribution.\n*/\n\n/** \\page maxipage Optimization\n\nThis section includes some notes on the maximum likelihood routine. As in the section\non writing models above, if a model has a \\c p or \\c log_likelihood method but no \\c\nestimate method, then calling \\c apop_estimate(your_data, your_model) executes the\ndefault estimation routine of maximum likelihood.\n\nIf you are a not a statistician, then there are a few things you will need to keep in\nmind:\n\n\\li Physicists, pure mathematicians, and the GSL minimize; economists, statisticians, and\nApophenia maximize. If you are doing a minimization, be sure that your function returns minus the objective\nfunction's value.\n\n\\li The overall setup is about estimating the parameters of a model with data. The user\nprovides a data set and an unparameterized model, and the system tries parameterized\nmodels until one of them is found to be optimal. The data is fixed. The optimization\ntries a series of parameterized models, searching for the one that is most likely. In\na non-stats setting, you may have \\c NULL data.\n\n\\li Because the unit of analysis is a parameterized model,\nnot just parameters, you need to have an \\ref apop_model\nwrapping your objective function.\n\nThis example, to be discussed in detail below, optimizes \nRosenbrock's banana function, \\f$(1-x)^2+ s(y - x^2)^2\\f$, where the\nscaling factor \\f$s\\f$ is fixed ahead of time, say at 100.\n\n\\include ../eg/banana.c\n\nThe \\c banana function returns a single number to be minimized. You will need to write an\n\\ref apop_model to send to the optimizer, which is a two step process: write a log\nlikelihood function wrapping the real objective function (\\c ll), and a model that uses that\nlog likelihood (\\c b).\n\n \\li The .vsize=2 part of the declaration of \\c b on the second\nline of main() specified that the model's parameters are a vector of\nsize two. That is, the list of doubles to send to \\c banana is set in\nin->parameters->vector->data. \n \\li The \\c more element of the \\ref apop_model\nstructure is designed to hold any arbitrary structure of size \\c more_size, which\nis useful for models that require additional constants or other settings, like the\ncoeff_struct here. See \\ref settingswriting for more on handling model settings.\n \\li Statisticians want the covariance and basic tests about the parameters. \nThis line shuts off all auxiliary calculations:\n\\code\nApop_settings_add_group(your_model, apop_parts_wanted);\n\\endcode\nSee the documentation for \\ref apop_parts_wanted_settings for details about how this\nworks. It can also offer quite the speedup: especially for high-dimensional problems,\nfinding the covariance matrix without any additional information can take dozens of evaluations\nof the objective function for each evaluation that is part of the search itself.\n \\li MLEs have an especially large number of parameter tweaks that could be made;\nsee the \\ref apop_mle_settings page.\n \\li As a useful diagnostic, you can add a \\c NULL \\ref apop_data set to the MLE\nsettings in the .path slot, and it will be allocated and filled with the\nsequence of points tried by the optimizer.\n \\li The program has some extras above and beyond the necessary: it uses two methods\n(notice how easy it is to re-run an estimation with an alternate method, but the syntax\nfor modifying a setting differs from the initialization syntax) and checks that the\nresults are accurate.\n\n\n\\section constr Setting Constraints\n\nThe problem is that the parameters of a function must not take on certain values,\neither because the function is undefined for those values or because parameters with\ncertain values would not fit the real-world problem.\n\nIf you give the optimizer an unconstrained likelihood function plus a separate constraint\nfunction, \\ref apop_maximum_likelihood will combine them to a function that is continuous\nat the constraint boundary, but which is guaranteed to never have an optimum outside of the constraint.\n\nA constraint function must do three things:\n\\li If the constraint does not bind (i.e. the parameter values are OK), then it must return zero.\n\\li If the constraint does bind, it must return a penalty, that indicates how far off the parameter is from meeting the constraint.\n\\li If the constraint does bind, it must set a return vector that the likelihood function can take as a valid input. The penalty at this returned value must be zero.\n\nThe idea is that if the constraint returns zero, the log likelihood function will\nreturn the log likelihood as usual, and if not, it will return the log likelihood at\nthe constraint's return vector minus the penalty. To give a concrete example, here\nis a constraint function that will ensure that both parameters of a two-dimensional\ninput are both greater than zero, and that their sum is greater than two. As with the\nconstraints for many of the models that ship with Apophenia, it is a wrapper for \\ref\napop_linear_constraint.\n\n\\code\nstatic long double greater_than_zero_constraint(apop_data *data, apop_model *v){\n static apop_data *constraint = NULL;\n if (!constraint) constraint= apop_data_falloc((3,3,2), 0, 1, 0, //0 < 1x + 0y\n 0, 0, 1, //0 < 0x + 1y\n 2, 1, 1); //2 < 1x + 1y\n return apop_linear_constraint(v->parameters->vector, constraint, 1e-3);\n}\n\\endcode\n\n\\li\\ref apop_linear_constraint()\n\n\\section simanneal Notes on simulated annealing\n\nFor convex optimizations, methods like conjugate gradient search work well, and\nfor relatively smooth optimizations, the Nelder-Mead simplex algorithm is a good\nchoice. For situations where the surface being searched may have several local optima\nand be otherwise badly behaved, there is simulated annealing.\n\nSimulated annealing is a controlled random walk. As with the other methods, the\nsystem tries a new point, and if it is better, switches. Initially, the system is\nallowed to make large jumps, and then with each iteration, the jumps get smaller,\neventually converging. Also, there is some decreasing probability that if the new\npoint is less likely, it will still be chosen. Simulated annealing is best for\nsituations where there may be multiple local optima. Early in the random walk, the\nsystem can readily jump from one to another; later it will fine-tune its way toward the\noptimum. The number of points tested is determined by the parameters of the simulated\ncolling program, not the values returned by the likelihood function. If you know your\nfunction is globally convex (as are most standard probability functions), then this\nmethod is overkill.\n\n\n\\section mlfns Useful functions\n\n\\li\\ref apop_estimate_restart : Restarting an MLE with different settings can improve results.\n\\li\\ref apop_maximum_likelihood : Rarely called directly. If a model has no \\c estimate element, call \\ref apop_estimate to prep the model and run an MLE.\n\\li\\ref apop_model_numerical_covariance\n\\li\\ref apop_numerical_gradient\n*/\n\n\n/** \\page moreasst Assorted\n\nSome functions for missing data:\n\n\\li\\ref apop_data_listwise_delete\n\\li\\ref apop_ml_impute\n\nA few more descriptive methods:\n\n\\li\\ref apop_matrix_pca : Principal component analysis\n\\li\\ref apop_anova : One-way or two-way ANOVA tables\n\\li\\ref apop_rake : Iterative proportional fitting on large, sparse tables\n\n\nGeneral utilities:\n\n\\li\\ref Apop_stopif : Apophenia's error-handling and warning-printing macro. \n\\li\\ref apop_opts : the global options\n\\li\\ref apop_system : a printf-style wrapper around the standard \\c system function.\n\nA few more math utilities:\n\n\\li\\ref apop_matrix_is_positive_semidefinite\n\\li\\ref apop_matrix_to_positive_semidefinite\n\\li\\ref apop_generalized_harmonic\n\\li\\ref apop_multivariate_gamma\n\\li\\ref apop_multivariate_lngamma\n\\li\\ref apop_rng_alloc\n\n*/\n\n/** \\page mingw MinGW\n\nMinimalist GNU for Windows is indeed minimalist: it is not a full POSIX subsystem, and provides no package manager. Therefore, you will have to make some adjustments and install the dependencies yourself.\n\nMatt P. Dziubinski successfully used Apophenia via MinGW; here are his instructions (with edits by BK):\n\n\\li get libregex (the ZIP file) from:\nhttp://sourceforge.net/project/showfiles.php?group_id=204414&package_id=306189\n\\li get libintl (three ZIP files) from:\n http://gnuwin32.sourceforge.net/packages/libintl.htm .\n download \"Binaries\", \"Dependencies\", \"Developer files\"\n\\li follow \"libintl\" steps from:\nhttp://kayalang.org/download/compiling/windows\n\n\\li Modify \\c Makefile, adding -lpthread to AM_CFLAGS (removing -pthread) and -lregex to AM_CFLAGS and LIBS\n\n\\li Now compile the main library:\n\\code\nmake\n\\endcode\n\n\\li Finally, put one more expected directory in place and install:\n\\code\nmkdir -p -- \"/usr/local/Lib/site-packages\"\nmake install\n\\endcode\n\n\\li You will get the usual warning about library paths, and may have to take the specified action:\n\\code\n----------------------------------------------------------------------\nLibraries have been installed in:\n /usr/local/lib\n\nIf you ever happen to want to link against installed libraries\nin a given directory, LIBDIR, you must either use libtool, and\nspecify the full pathname of the library, or use the `-LLIBDIR'\nflag during linking and do at least one of the following:\n - add LIBDIR to the `PATH' environment variable\n during execution\n - add LIBDIR to the `LD_RUN_PATH' environment variable\n during linking\n - use the `-LLIBDIR' linker flag\n\nSee any operating system documentation about shared libraries for\nmore information, such as the ld(1) and ld.so(8) manual pages.\n----------------------------------------------------------------------\n\\endcode\n*/\n\n\n/* optionaldetails Implementation of optional arguments [this section ignored by doxygen]\nOptional and named arguments are among the most commonly commented-on features of Apophenia, so this page goes into full detail about the implementation. \n\nTo use these features, see the all-you-really-need summary at the \\ref designated\npage. For a background and rationale, see the blog entry at http://modelingwithdata.org/arch/00000022.htm . \n\nI'll assume you've read both links before continuing.\n\nOK, now that you've read the how-to-use and the discussion of how optional and named arguments can be constructed in C, this page will show how they are done in Apophenia. The level of details should be sufficient to implement them in your own code if you so desire.\n\nThere are three components to the process of generating optional arguments as implemented here:\n\\li Produce a \\c struct whose elements match the arguments to the function.\n\\li Write a wrapper function that takes in the struct, unpacks it, and calls the original function.\n\\li Write a macro that makes the user think the wrapper function is the real thing.\n\nNone of these steps are really rocket science, but there is a huge amount of redundancy. \nApophenia includes some macros that reduce the boilerplate redundancy significantly. There are two layers: the C-standard code, and the script that produces the C-standard code.\n\nWe'll begin with the C-standard header file:\n\\code \n#ifdef APOP_NO_VARIADIC\n void apop_vector_increment(gsl_vector * v, int i, double amt);\n#else\n void apop_vector_increment_base(gsl_vector * v, int i, double amt);\n apop_varad_declare(void, apop_vector_increment, gsl_vector * v; int i; double amt);\n#define apop_vector_increment(...) apop_varad_link(apop_vector_increment, __VA_ARGS__)\n#endif\n\\endcode\n\nFirst, there is an if/else that allows the system to degrade gracefully\nif you are sending C code to a parser like swig, whose goals differ\ntoo much from straight C compilation for this to work. Set \\c\nAPOP_NO_VARIADIC to produce a plain function with no variadic support.\n\nElse, we begin the above steps. The \\c apop_varad_declare line expands to the following:\n\n\\code\ntypedef struct { \n gsl_vector * v; int i; double amt ; \n} variadic_type_apop_vector_increment; \n\nvoid variadic_apop_vector_increment(variadic_type_apop_vector_increment varad_in);\n \\endcode\n\nSo there's the ad-hoc struct and the declaration for the wrapper\nfunction. Notice how the arguments to the macro had semicolons, like a\nstruct declaration, rather than commas, because the macro does indeed\nwrap the arguments into a struct.\n\n Here is what the \\c apop_varad_link would expand to:\n \\code\n#define apop_vector_increment(...) variadic_apop_increment_base((variadic_type_apop_vector_increment) {__VA_ARGS__})\n \\endcode\nThat gives us part three: a macro that lets the user think that they are\nmaking a typical function call with a set of arguments, but wraps what\nthey type into a struct.\n\nNow for the code file where the function is declared. Again, there is is an \\c APOP_NO_VARIADIC wrapper. Inside the interesting part, we find the wrapper function to unpack the struct that comes in.\n\n\\code\n\\#ifdef APOP_NO_VARIADIC \n void apop_vector_increment(gsl_vector * v, int i, double amt){\n\\#else\napop_varad_head( void , apop_vector_increment){\n gsl_vector * apop_varad_var(v, NULL);\n Apop_stopif(!v, return, 0, \"You sent me a NULL vector.\");\n int apop_varad_var(i, 0);\n double apop_varad_var(amt, 1);\n apop_vector_increment_base(v, i, amt);\n}\n\n void apop_vector_increment_base(gsl_vector * v, int i, double amt){\n#endif\n\tv->data[i * v->stride]\t+= amt;\n}\n\\endcode\n\nThe \n\\c apop_varad_head macro reduces redundancy, and will expand to\n\\code\nvoid variadic_apop_vector_increment (variadic_type_variadic_apop_vector_increment varad_in)\n\\endcode\n\nThe function with this header thus takes in a single struct, and for every variable, there is a line like\n\\code\n double apop_varad_var(amt, 1);\n\\endcode\nwhich simply expands to:\n\\code\n double amt = varad_in.amt ? varad_in.amt : 1;\n\\endcode\nThus, the macro declares each not-in-struct variable, and so there will need to be\none such declaration line for each argument. Apart from requiring declarations, you\ncan be creative: include sanity checks, post-vary the variables of the inputs, unpack\nwithout the macro, and so on. That is, this parent function does all of the bookkeeping,\nchecking, and introductory shunting, so the base function can do the math. Finally,\nthe introductory section will call the base function.\n\nThe setup goes out of its way to leave the \\c _base function in the public namespace,\nso that those who would prefer speed to bounds-checking can simply call that function\ndirectly, using standard notation. You could eliminate this feature by merging\nthe two functions.\n\n\nThe m4 script\n\nThe above is all you need to make this work: the varad.h file, and the above structures. But there is still a lot of redundancy, which can't be eliminated by the plain C preprocessor.\n\nThus, in Apophenia's code base (the one you'll get from checking out the git repository, not the gzipped distribution that has already been post-processed) you will find a pre-preprocessing script that converts a few markers to the above form. Here is the code that will expand to the above C-standard code:\n\n\\code\n//header file\nAPOP_VAR_DECLARE void apop_vector_increment(gsl_vector * v, int i, double amt);\n\n//code file\nAPOP_VAR_HEAD void apop_vector_increment(gsl_vector * v, int i, double amt){\n gsl_vector * apop_varad_var(v, NULL);\n Apop_stopif(!v, return, 0, \"You sent me a NULL vector.\");\n int apop_varad_var(i, 0);\n double apop_varad_var(amt, 1);\nAPOP_VAR_END_HEAD\n\tv->data[i * v->stride]\t+= amt;\n}\n\\endcode\n\nIt is obviously much shorter. The declaration line is actually a C-standard declaration with the \\c APOP_VAR_DECLARE preface, so you don't have to remember when to use semicolons. The function itself looks like a single function, but there is again a marker before the declaration line, and the introductory material is separated from the main matter by the \\c APOP_VAR_END_HEAD line. Done right, drawing a line between the introductory checks or initializations and the main function can improve readability.\n\nThe m4 script inserts a return function_base(...) at the end of the header\nfunction, so you don't have to. If you want to call the function before the last line, you\ncan do so explicitly, as in the expansion above, and add a bare return; to\nguarantee that the call to the base function that the m4 script will insert won't ever be\nreached.\n\nOne final detail: it is valid to have types with commas in them---function arguments. Because commas get turned to semicolons, and m4 isn't a real parser, there is an exception built in: you will have to replace commas with exclamation marks in the header file (only). E.g.,\n\n\\code\nAPOP_VAR_DECLARE apop_data * f_of_f(apop_data *in, void *param, int n, double (*fn_d)(double ! void * !int));\n\\endcode\n\nm4 is POSIX standard, so even if you can't read the script, you have the program needed to run it. For example, if you name it \\c prep_variadics.m4, then run\n\\code\nm4 prep_variadics.m4 myfile.m4.c > myfile.c\n\\endcode\n*/\n\n\n/**\n\\page gentle A quick overview\n\nThis is a \"gentle introduction\" to the Apophenia library. It is intended \nto give you some initial bearings on the typical workflow and the concepts and tricks that\nthe manual pages assume you are familiar with.\n\nIf you want to install Apophenia now so you can try the samples on this page, see the \\ref setup page.\n\nAn outline of this overview:\n\n\\li Apophenia fills a space between traditional C libraries and stats packages. \n\\li The \\ref apop_data structure represents a data set (of course). Data sets are inherently complex,\nbut there are many functions that act on \\ref apop_data sets to make life easier.\n\\li The \\ref apop_model encapsulates the sort of actions one would take with a model, like estimating model parameters or predicting values based on new inputs.\n\\li Databases are great, and a perfect fit for the sort of paradigm here. Apophenia\nprovides functions to make it easy to jump between database tables and \\ref apop_data sets.\n\n The opening example\n\nSetting aside the more advanced applications and model-building tasks, let us begin with\nthe workflow of a typical fitting-a-model project using Apophenia's tools:\n\n\\li Read the raw data into the database using \\ref apop_text_to_db.\n\\li Use SQL queries handled by \\ref apop_query to massage the data as needed.\n\\li Use \\ref apop_query_to_data to pull some of the data into an in-memory \\ref apop_data set.\n\\li Call a model estimation such as \\code apop_estimate (data_set, apop_ols)\\endcode or \\code apop_estimate (data_set, apop_probit)\\endcode to fit parameters to the data. This will return an \\ref apop_model with parameter estimates.\n\\li Interrogate the returned estimate, by dumping it to the screen with \\ref apop_model_print, sending its parameters and variance-covariance matrices to additional tests (the \\c estimate step runs a few for you), or send the model's output to be input to another model.\n\nHere is an example of most of the above steps which you can compile and run, to be discussed in detail below.\n\nThe program relies on the U.S. Census's American Community Survey public use microdata for DC 2008, which you can get from the command line via:\n\n\\code\nwget https://raw.github.com/rodri363/tea/master/demo/ss08pdc.csv\n\\endcode\nor by pointing your browser to that address and saving the file.\n\nThe program:\n\\code\n#include \n\nint main(){\n apop_text_to_db(.text_file=\"ss08pdc.csv\", .tabname=\"dc\");\n apop_data *data = apop_query_to_data(\"select log(pincp+10) as log_income, agep, sex \"\n \"from dc where agep+ pincp+sex is not null and pincp>=0\");\n apop_model *est = apop_estimate(data, apop_ols);\n apop_model_print(est);\n}\n\\endcode\n\nIf you saved the code to census.c and don't have a \\ref makefile or other\nbuild system, then you can compile it with\n\n\\code\ngcc census.c -std=gnu99 -lapophenia -lgsl -lgslcblas -lsqlite3 -o census\n\\endcode\n\nor \n\n\\code\nclang census.c -lapophenia -lgsl -lgslcblas -lsqlite3 -o census\n\\endcode\n\nand then run it with ./census. This compile line will work on any system with all the requisite tools,\nbut for full-time work with this or any other C library, you will probably want to write a \\ref makefile.\n\nThe results are unremarkable---age has a positive effect on income, and sex\n(1=male, 2=female) does has a negative effect---but it does give us some lines of\ncode to dissect.\n\nThe first two lines in \\c main() make use of a database. \nI'll discuss the value of the database step more at the end of this page, but for now,\nnote that there are several functions, \\ref apop_query and \\ref apop_query_to_data\nbeing the ones you will most frequently be using, that will allow you to talk to and\npull data from either an SQLite or mySQL/mariaDB database. The database is a natural\nplace to do data processing like renaming variables, selecting subsets, and transforming values.\n\n Designated initializers\n\nLike this line,\n\n\\code\napop_text_to_db(.text_file=\"data\", .tabname=\"d\");\n\\endcode\n\nmany Apophenia functions accept named, optional arguments. To give another example,\nthe \\ref apop_data set has the usual row and column numbers, but also row and column\nnames. So you should be able to refer to a cell by any combination of name or number;\nfor the data set you read in above, which has column names, all of the following work:\n\n\\code\nx = apop_data_get(data, 2, 3); //observation 2, column 3\nx = apop_data_get(data, .row=2, .colname=\"sex\"); // same\napop_data_set(data, 2, 3, 1);\napop_data_set(data, .colname=\"sex\", .row=2, .val=1);\n\\endcode\n\nDefault values mean that the \\ref apop_data_get, \\ref apop_data_set, and \\ref apop_data_ptr functions handle matrices, vectors, and scalars sensibly:\n\\code\n//Let v be a hundred-element vector:\napop_data *v = apop_data_alloc(100);\n[fill with data here]\ndouble x1 = apop_data_get(v, 10);\napop_data_set(v, 2, .val=x1);\n\n//A 100x1 matrix behaves like a vector\napop_data *m = apop_data_alloc(100, 1);\n[fill with data here]\ndouble m1 = apop_data_get(v, 1);\n\n//let s be a scalar stored in a 1x1 apop_data set:\napop_data *v = apop_data_alloc(1);\ndouble *scalar = apop_data_ptr(s);\n\\endcode\n\nThese conveniences may be new to users of less user-friendly C libraries, but it it fully\nconforms to the C standard (ISO/IEC 9899:2011). See the \\ref designated page for details.\n\n\n\\section apop_data\n\nA lot of real-world data processing is about quotidian annoyances about text versus\nnumeric data or dealing with missing values, and the \\ref apop_data set and its\nmany support functions are intended to make data processing in C easy. Some users of\nApophenia use the library only for its \\ref apop_data set and associated functions. See\n\\ref dataoverview for extensive notes on using the structure.\n\nThe structure includes seven parts:\n\n\\li a vector,\n\\li a matrix,\n\\li a grid of text elements,\n\\li a vector of weights,\n\\li names for everything: row names, a vector name, matrix column names, text names,\n\\li a link to a second page of data, and\n\\li an error marker\n\nThis is not a generic and abstract ideal, but is the sort of mess that real-world data sets look like. For\nexample, here is some data for a weighted OLS regression. It includes an outcome\nvariable in the vector, dependent variables in the matrix and text grid,\nreplicate weights, and column names in bold labeling the variables:\n\n\\htmlinclude apop_data_fig.html\n\\latexinclude apop_data_fig.tex\n\nApophenia's functions generally assume that one row across all of these elements\ndescribes a single observation or data point.\n\nSee above for some examples of getting and setting individual elements.\n\nAlso, \\ref apop_data_get, \\ref apop_data_set, and \\ref apop_data_ptr consider the vector to be the -1st column,\nso using the data set in the figure, apop_data_get(sample_set, .row=0, .col=-1) == 1.\n\n Reading in data\n\nAs per the example above, use \\ref apop_text_to_data or \\ref apop_text_to_db and then \\ref apop_query_to_data.\n\n Subsets\n\nThere are many macros to get views of subsets of the data. Each generates a disposable\nwrapper around the base data: once the variable goes out of scope, the wrapper\ndisappears, but modifications made to the data in the view are modifications to the\nbase data itself.\n\n\\include simple_subsets.c\n\nAll of these slicing routines are macros, because they generate several\nbackground variables in the current scope (something a function can't do). Traditional\ncustom is to put macro names in all caps, like \\c APOP_DATA_ROWS, which to modern\nsensibilities looks like yelling. The custom has a logic: there are ways to hang\nyourself with macros, so it is worth distinguishing them typographically. \nApophenia tones it down by capitalizing only the first letter.\n\n Basic manipulations\n\nSee \\ref dataoverview for a list of many other manipulations of data sets, such as \n\\ref apop_data_listwise_delete for quick-and-dirty removal of observations with NaNs,\n\\ref apop_data_split / \\ref apop_data_stack,\nor \\ref apop_data_sort to sort all elements by a single column.\n\n Apply and map\n\nIf you have an operation of the form for each element of my data set, call this\nfunction, then you can use \\ref apop_map to do it. You could basically do everything you\ncan do with an apply/map function via a \\c for loop, but the apply/map approach is clearer\nand more fun. Also, if you set OpenMP's omp_set_num_threads(N) for any \\c N\ngreater than 1 (the default on most systems is the number of CPU cores), then the work\nof mapping will be split across multiple CPU threads. See \\ref mapply for a number\nof examples.\n\n Text\n\nString handling in C usually requires some tedious pointer and memory handling, but the functions \nto put strings into the text grid in the \\ref apop_data structure and get them out\nagain will do the pointer shunting for you. The \\ref apop_text_alloc function is\nreally a realloc function: you can use it to resize the text grid as necessary. The\n\\ref apop_text_set function will write a single string to the grid, though you may be\nusing \\ref apop_query_to_text or \\ref apop_query_to_mixed_data to read in an entire\ndata set at once. Functions that act on entire data sets, like \\ref apop_data_rm_rows,\nhandle the text part as well.\n\nThe text grid for \\c your_data has your_data->textsize[0] rows and your_data->textsize[1] columns. If you are using only the functions to this point, then empty elements are a blank string (\"\"), not \\c NULL.\nFor reading individual elements, refer to the \\f$(i,j)\\f$th text element via your_data->text[i][j].\n\n Errors\n\nMany functions will set the error element of the \\ref apop_data structure being operated on if anything goes wrong. You can use this to halt the program or take corrective action:\n\n\\code \napop_data *the_data = apop_query_to_data(\"select * from d\");\nApop_stopif(!the_data || the_data->error, exit(1), 0, \"Trouble querying the data\");\n\\endcode \n\n The whole structure\n\nHere is a diagram of all of Apophenia's structures and how they\nrelate. It is taken from this\ncheat sheet on general C and SQL use (2 page PDF).\n\n\\image html http://apophenia.info/structs.png width=\"100%\"\n\\image latex ../structs.png width=18cm\n\nAll of the elements of the \\ref apop_data structure are laid out at middle-left. You have\nalready met the vector, matrix, weights, and text grid.\n\nThe diagram shows the \\ref apop_name structure, which has received little mention so far because names\nbasically take care of themselves. A query will bring in column names (and row names if you set apop_opts.db_name_column), or use \\ref apop_data_add_names to add names to your data\nset and \\ref apop_name_stack to copy from one data set to another.\n\nThe \\ref apop_data structure has a \\c more element, for when your data is best expressed\nin more than one page of data. Use \\ref apop_data_add_page, \\ref apop_data_rm_page,\nand \\ref apop_data_get_page. Output routines will sometimes append an extra page of\nauxiliary information to a data set, such as pages named \\ or\n\\. The angle-brackets indicate a page that describes the data set\nbut is not a part of it (so an MLE search would ignore that page, for example).\n\n\nNow let us move up the structure diagram to the \\ref apop_model structure. \n\n\\section gentle_model apop_model\n\nEven restricting ourselves to the most basic operations, there are a lot of things that \nwe want to do with our models: use a data set to estimate the parameters of a model (like the mean and\nvariance of a Normal distribution), or draw random numbers, or show the\nexpected value, or show the expected value of one part of the data given fixed values\nfor the rest of it. The \\ref apop_model is intended to encapsulate most of these desires\ninto one object, so that models can easily be swapped around, modified to create new models, \ncompared, and so on.\n\nFrom the figure above, you can see that the \\ref apop_model structure \nincludes a number of informational items, key being the \\c parameters, \\c data, and\n\\c info elements; a list of settings to be discussed below; and a set of procedures\nfor many operations. Its contents are not (entirely) arbitrary: the overall intent\nand the theoretical basis for what is and is not included in an \\ref apop_model are\ndescribed in this U.S.\nCensus Bureau research report.\n\nThere are helper functions that will allow you to avoid dealing with the model\ninternals. For example, the \\ref apop_estimate helper function means you never have\nto look at the model's \\c estimate method (if it even has one), and you will simply\npass the model to a function, as with the above form:\n\n\\code\n apop_model *est = apop_estimate(data, apop_ols);\n\\endcode\n\n\\li Apophenia ships with a broad set of models, like \\ref apop_ols, \\ref apop_dirichlet,\n \\ref apop_loess, and \\ref apop_pmf (probability mass function); see the full list on the models documentation page. You would fit\nany of them using \\ref apop_estimate call, with the appropriate model as the second input.\n\\li The models that ship with Apophenia, like \\ref apop_ols, include the procedures and some metadata, but are of course not yet estimated using a data set (i.e., data == NULL, parameters == NULL). The line above generated a new\nmodel, \\c est, which is identical to the base OLS model but has estimated parameters\n(and covariances, and basic hypothesis tests, a log likelihood, \\f$AIC_c\\f$, \\f$BIC\\f$, et cetera), and a \\c data pointer to the \\ref apop_data set used for estimation. \n\\li You will mostly use the models by passing them as inputs to \nfunctions like \\ref apop_estimate, \\ref apop_draw, or \\ref apop_predict; more examples below.\nOther than \\ref apop_estimate, most require a parameterized model like \\c est. After all, it doesn't make sense to\ndraw from a Normal distribution until its mean and standard deviation are specified.\n\\li If you know what the parameters should be, for most models use \\ref apop_model_set_parameters. E.g.\n\n\\code\napop_model *std_normal = apop_model_set_parameters(apop_normal, 0, 1);\napop_data *a_thousand_normals = apop_model_draws(std_normal, 1000);\n\napop_model *poisson = apop_model_set_parameters(apop_poisson, 1.5);\napop_data *a_thousand_waits = apop_model_draws(poisson, 1000);\n\\endcode\n\n\\li You can use \\ref apop_model_print to print the various elements to screen.\n\\li You can combine and transform models with functions such as \\ref\napop_model_fix_params, \\ref apop_model_coordinate_transform, or \\ref\napop_model_mixture. Each of these functions produce a new model, which can be estimated,\nre-combined, or otherwise used like any other model.\n\n\\code\n//A helper function to check whether a data point is nonnegative\ndouble over_zero(apop_data *in, apop_model *m){ return apop_data_get(in) > 0; }\n\n//Generate a truncated Normal distribution by adding a data constraint:\napop_model *truncated_normal= apop_model_dconstrain(.base_model=apop_normal,\n .constraint=over_zero);\n\n//Get the cross product of that and a free Normal.\napop_model *cross = apop_model_cross(apop_normal, truncated_normal);\n\n//Given assumed data, estimate the parameters of the cross product.\napop_model *xest = apop_estimate(assumed_data, cross);\n\n//Assuming more data, use the fitted cross product as the prior for a Normal distribution.\napop_model *posterior = apop_update(moredata, .prior=xest, .likelihood=apop_normal);\n\n//Assuming more data, use the posterior as the prior for another updating round.\napop_model *post2 = apop_update(moredata2, .prior=posterior, .likelihood=apop_normal);\n\\endcode\n\n\\li Writing your own models won't be covered in this introduction, but it can be easy to\ncopy and modify the procedures of an existing model to fit your needs. When in doubt, delete a procedure, because any procedures that are missing will have\ndefaults filled when used by functions like \\ref apop_estimate (which uses \\ref\napop_maximum_likelihood) or \\ref apop_cdf (which uses integration via random draws). See \\ref modeldetails for details.\n\\li There's a simple rule of thumb for remembering the order of the arguments to most of\nApophenia's functions, including \\ref apop_estimate : the data always comes first.\n\n Settings\n\nHow many bins are in a histogram? At what tolerance does the maximum likelihood\nsearch end? What are the models being combined in an \\ref apop_mixture distribution?\n\nApophenia organizes settings in settings groups, which are then attached\nto models. In the following snippet demonstrating Bayesian updating, we specify a Beta distribution prior. If the\nlikelihood function were a Binomial distribution, \\ref apop_update knows the closed-form\nposterior for a Beta-Binomial pair, but in this case, with a PMF as a likelihood,\nit will have to run Markov chain Monte Carlo. The \\ref apop_mcmc_settings group attached\nto the prior specifies details of how the run should work.\n\nFor a likelihood, we generate an empirical distribution---a PMF---from an input \ndata set, via apop_estimate(your_data, apop_pmf).\nWhen we call \\ref apop_update on the last line, it already has all of the above info\non hand.\n\n\\code\napop_model *beta = apop_model_set_parameters(apop_beta, 0.5, 0.25);\nApop_settings_add_group(beta, apop_mcmc, .burnin = 0.2, .periods =1e5);\napop_model *my_pmf = apop_estimate(your_data, apop_pmf);\napop_model *posterior = apop_update(.prior= beta, .likelihood = my_pmf);\n\\endcode\n\n Databases and models\n\nReturning to the introductory example, you saw that (1) the\nlibrary expects you to keep your data in a database, pulling out the\ndata as needed, and (2) that the workflow is built around\n\\ref apop_model structures.\n\nStarting with (2), \nif a stats package has something called a model, then it is\nprobably of the form Y = [an additive function of X], such as \\f$y = x_1 +\n\\log(x_2) + x_3^2\\f$. Trying new models means trying different\nfunctional forms for the right-hand side, such as including \\f$x_1\\f$ in\nsome cases and excluding it in others. Conversely, Apophenia is designed \nto facilitate trying new models in the broader sense of switching out a \nlinear model for a hierarchical, or a Bayesian model for a simulation. \nA formula syntax makes little sense over such a broad range of models.\n\nAs a result, the right-hand side is not part of \nthe \\ref apop_model. Instead, the data is assumed to be correctly formatted, scaled, or logged\nbefore being passed to the model. This is where part (1), the database,\ncomes in, because it provides a proxy for the sort of formula specification language above:\n \\code\napop_data *testme= apop_query_to_data(\"select y, x1, log(x2), pow(x3, 2) from data\");\napop_model *est = apop_estimate(testme, apop_ols);\n\\endcode\n\nGenerating factors and dummies is also considered data prep, not model\ninternals. See \\ref apop_data_to_dummies and \\ref apop_data_to_factors.\n\nNow that you have \\c est, an estimated model, you can interrogate it. This is where Apophenia and its encapsulated\nmodel objects shine, because you can do more than just admire the parameter estimates on\nthe screen: you can take your estimated data set and fill in or generate new data, use it\nas an input to the parent distribution of a hierarchical model, et cetera. Some simple\nexamples:\n\n \\code\n //If you have a new data set with missing elements (represented by NaN), you can fill in predicted values:\napop_predict(new_data_set, est);\napop_data_print(new_data_set);\n\n //Fill a matrix with random draws.\napop_data *d = apop_model_draws(est, .count=1000);\n\n //How does the AIC_c for this model compare to that of est2?\nprintf(\"ΔAIC_c=%g\\n\", apop_data_get(est->info, .rowname=\"AIC_c\") \n - apop_data_get(est2->info, .rowname=\"AIC_c\"));\n\\endcode\n\n\\section gentle_end Conclusion\n\nThis introduction has shown you the \\ref apop_data set and some of the functions\nassociated, which might be useful even if you aren't formally doing statistical\nwork but do have to deal with data with real-world elements like column names and\nmixed numeric/text values. You've seen how Apophenia encapsulates many of a model's\ncharacteristics into a single \\ref apop_model object, which you can send with data to\nfunctions like \\ref apop_estimate, \\ref apop_predict, or \\ref apop_draw. Once you've\ngot your data in the right form, you can use this to simply estimate model parameters,\nor as an input to later analysis.\n\nWhat's next? \n \\li Check out the system for hypothesis testing, both with traditional known\ndistributions (using \\ref apop_test for dealing with Normal-, \\f$t\\f$-,\n\\f$\\chi^2\\f$-distributed statistics); and for the parameters of any model; in \\ref testpage.\n \\li Try your own hand at putting new models into the \\ref apop_model framework,\nas discussed in \\ref modeldetails.\n \\li For example, have a look at this blog\nand its subsequent posts, which wrap a microsimulation into an \\ref apop_model, so\nthat its parameters can be estimated and confidence intervals set around them.\n \\li See the \\ref maxipage page for discussion of the many features the optimization\nsystem has. It allows you to use a diverse set of search types on constrained or\nunconstrained models.\n \\li Skim through the full list\nof macros and functions---there are hundreds---to get a sense of what else\nApophenia offers.\n*/\n\n/** \\page modeldetails Writing new models\n\nThe \\ref apop_model is intended to provide a consistent expression of any\nmodel that (implicitly or explicitly) expresses a likelihood of data given parameters,\nincluding traditional linear models, textbook distributions, Bayesian hierarchies,\nmicrosimulations, and any combination of the above. The unifying feature is that\nall of the models act over some data space and some parameter space (in some cases\none or both is the empty set), and can assign a likelihood for a fixed pair of\nparameters and data given the model. This is a very broad requirement, often used\nin the statistical literature. For discussion of the theoretical structures, see A Useful Algebraic System\nof Statistical Models (PDF).\n\nThis page is about writing new models from scratch, beginning with basic models and on\nup to models with arbitrary internal settings, specific methods of Bayesian updating\nusing your model as a prior or likelihood, and so on. I assume you have already read\n\\ref modelsec on using models and have tried a few things with the\ncanned models that come with Apophenia, so you already know how a user handles basic\nestimation, adding a settings group, and so on.\n\nThis page includes:\n\n\\li \\ref write_likelihoods of writing a new model from scratch.\n\\li \\ref settingswriting, covering the writing of ad hoc structures to hold model- or method-specific details, like the number of periods for burning in an MCMC run or the number of bins in a histogram.\n\\li \\ref vtables, covering the means of writing special-case routines for functions that are not part of the \\ref apop_model itself, including the score or conjugate prior/likelihood pairs for \\ref apop_update.\n\\li \\ref modeldataparts, a detailed list of the requirements for the non-function elements of an \\ref apop_model.\n\\li \\ref methodsection, a detailed list of requirements for the function elements of an \\ref apop_model.\n\n\\section write_likelihoods A walkthrough\n\nUsers are encouraged to always use models via the helper functions, like\n\\ref apop_estimate or \\ref apop_cdf. The helper functions do some boilerplate error\nchecking, and call defaults as needed. For example, if your model has a \\c log_likelihood\nmethod but no \\c p method, then \\ref apop_p will use exp(\\c log_likelihood). If you don't\ngive an \\c estimate method, then \\c apop_estimate will call \\ref apop_maximum_likelihood.\n\nSo the game in writing a new model is to write just enough internal methods to give the helper functions what they need.\nIn the not-uncommon best case, all you need to do is write a log likelihood function or an RNG.\n\nHere is how one would set up a model that could be estimated using maximum likelihood:\n\n\\li Write a likelihood function. Its header will look like this:\n\\code\nlong double new_log_likelihood(apop_data *data, apop_model *m);\n\\endcode\nwhere \\c data is the input data, and \\c m is the parametrized model (i.e. your model\nwith a \\c parameters element already filled in by the caller). This function will\nreturn the value of the log likelihood function at the given parameters.\n\n\\li Write the object:\n\n\\code\napop_model *your_new_model = &(apop_model){\"The Me distribution\", \n .vsize=n0, .msize1=n1, .msize2=n2, .dsize=nd,\n .log_likelihood = new_log_likelihood };\n\\endcode\n\n \\li The first element is the .name, a human-language name for your model.\n \\li The \\c vsize, \\c msize1, and \\c msize2 elements specify the shape of the parameter\nset. For example, if there are three numbers in the vector, then set .vsize=3\nand omit the matrix sizes. The default model prep routine will call\nnew_est->parameters = apop_data_alloc(vsize, msize1, msize2). \n \\li The \\c dsize element is the size of one random draw from your model.\n \\li It's common to have [the number of columns in your data set] parameters; this\ncount will be filled in if you specify \\c -1 for \\c vsize, msize(1|2), or\ndsize. If the allocation is exceptional in a different way, then you will\nneed to allocate parameters by writing a custom \\c prep method for the model.\n \\li Is this a constrained optimization? Add a .constraint element for those too. See \\ref constr for more.\n\nYou already have more than enough that something like this will work (the \\c dsize is used for random draws):\n\\code\napop_model *estimated = apop_estimate(your_data, your_new_model);\n\\endcode\n\nOnce that baseline works, you can fill in other elements of the \\ref apop_model as needed.\n\nFor example, if you are using a maximum likelihood method to estimate parameters, you can get much faster estimates and better covariance estimates by specifying the dlog likelihood function (aka the score):\n\n\\code\nvoid apop_new_dlog_likelihood(apop_data *d, gsl_vector *gradient, apop_model *m){\n //do algebra here to find df/dp0, df/dp1, df/dp2....\n gsl_vector_set(gradient, 0, d_0);\n gsl_vector_set(gradient, 1, d_1);\n}\n\\endcode\nThe score is not part of the model object, but is registered (see below) using \n\\code\napop_score_insert(apop_new_dlog_likelihood, your_new_model);\n\\endcode\n\n\\subsection On Threading\nMany procedures in Apophenia use OpenMP to thread operations, so assume your functions are running in a threaded environment. If a method can not be threaded, wrap it in an OpenMP critical region. E.g.,\n\n\n\\code\n\nvoid apop_new_dlog_likelihood(apop_data *d, gsl_vector *gradient, apop_model *m){\n #pragma omp critical (newdlog)\n {\n //un-threadable algebra here\n }\n gsl_vector_set(gradient, 0, d_0);\n gsl_vector_set(gradient, 1, d_1);\n}\n\\endcode\n\n\\section settingswriting Writing new settings groups\n\nYour model may need additional settings or auxiliary information to function, which\nwould require associating a model-specific struct with the model.\nA method associated with a model that uses such a struct usually begins with calls like\n\\code\nlong double ysg_ll(apop_data *d, apop_model *m){\n ysg_settings *sets = apop_settings_get(m, ysg);\n\n ...\n}\n\\endcode\n\nThese model-specific structs are handled as expected by \\ref apop_model_copy and \\ref\napop_model_free, and many functions that modify or transform an \\ref apop_model try to\nhandle settings groups as expected. This section describes how to build a settings\ngroup so all these automatic steps happen as expected, and your methods can reliably retrieve settings as needed.\n\nBut before getting into the detail of how to make model-specific groups of settings\nwork, note that there's a lightweight method of storing sundry settings, so in many\ncases you can bypass all of the following.\nThe \\ref apop_model structure has a \\c void pointer named \\c more which you can use to\npoint to a model-specific struct. If \\c more_size is larger than zero (i.e. you set\nit to your_model.more_size=sizeof(your_struct)), then it will be copied via \\c\nmemcpy by \\ref apop_model_copy, and freed by \\ref apop_model_free. Apophenia's\nroutines will never impinge on this item, so do what you wish with it.\n\nThe remainder of this subsection describes the information you'll have to provide to make\nuse of the conveniences described to this point: initialization of defaults, smarter\ncopying and freeing, and adding to an arbitrarily long list of settings groups attached\nto a model. You will need four items: a typedef for the structure itself, plus init, copy, and\nfree functions. This is the sort of boilerplate that will be familiar to users of\nobject-oriented languages in the style of C++ or Java, but it's really a list of\narbitrarily-typed elements, which makes this feel more like LISP. [And being a\nreimplementation of an existing feature of LISP, this section will be macro-heavy.]\n\nThe settings struct will likely go into a header file, so here is a sample header\nfor a new settings group named \\c ysg_settings, with a dataset, two sizes, and\na reference counter. ysg stands for Your Settings Group; replace that\nsubstring with your preferred name in every instance to follow.\n\n\\code\ntypedef struct {\n int size1, size2;\n char *refs;\n apop_data *dataset;\n} ysg_settings;\n\nApop_settings_declarations(ysg)\n\\endcode\n\nThe first item is a familiar structure definition. The last line is a macro that declares the\ninit, copy, and free functions discussed below. This is everything you would\nneed in a header file, should you need one. These are just declarations; we'll write\nthe actual init/copy/free functions below.\n\nThe structure itself gets the full name, \\c ysg_settings. Everything else is a macro\nkeyed on \\c ysg, without the \\c _settings part. Because of these macros, your struct\nname must end in \\c _settings.\n\nIf you have an especially simple structure, then you can generate the three functions with these three macros in your .c file:\n\n\\code\nApop_settings_init(ysg, )\nApop_settings_copy(ysg, )\nApop_settings_free(ysg, )\n\\endcode\n\nThese macros generate appropriate functions to do what you'd expect: allocating the\nmain structure, copying one struct to another, freeing the main structure. \nThe spaces after the commas indicate that in these cases no special code gets added to\nthe functions that these macros expand into.\n\nYou'll never call the generated functions directly; they are called by \\ref Apop_settings_add_group,\n\\ref apop_model_free, and other model or settings-group handling functions.\n\nNow that initializing/copying/freeing of\nthe structure itself is handled, the remainder of this section will be about how to\nadd instructions for the structure internals, like data that is pointed to by the structure elements.\n\n\\li For the allocate function, use the above form if everything in your code defaults to zero/\\c NULL. \nOtherwise, you will need a new line declaring a default for every element in your structure. There is a macro to help with this too. \nThese macros will define for your use a structure named \\c in, and an output pointer-to-struct named \\c out.\nContinuing the above example:\n\\code\nApop_settings_init (ysg, \n Apop_stopif(!in.size1, return NULL, 0, \"I need you to give me a value for size1.\");\n Apop_varad_set(size2, 10);\n Apop_varad_set(dataset, apop_data_alloc(out->size1, out->size2));\n Apop_varad_set(refs, malloc(sizeof(int)));\n *refs=1;\n)\n\\endcode\nNow, Apop_settings_add(a_model, ysg, .size1=100) would set up a group with\na 100-by-10 data set, and set the reference counter allocated and to one.\n\n\\li Some functions do extensive internal copying, so you will need a copy function even\nif your code has no explicit calls to \\ref apop_model_copy. The default above simply\ncopies every element in the structure. Pointers are copied, giving you two pointers\npointing to the same data. We have to be careful to prevent double-freeing later.\n\\code\n//The elements of the set to copy are all copied by the function's boilerplate,\n//and then make one additional modification:\nApop_settings_copy (ysg,\n #pragma omp critical (ysg_refs)\n (*refs)++;\n)\n\\endcode\n\n\\li The struct itself is freed by boilerplate code, but add code in the free function\nto free data pointed to by pointers in the main structure. The macro defines a\npointer-to-struct named \\c in for your use. Continuing the example:\n\\code\nApop_settings_free (ysg,\n #pragma omp critical (ysg_refs)\n if (!(--in->refs)) {\n free(in->dataset);\n free(in->refs);\n }\n)\n\\endcode\n\nWith those three macros in place and the header as above, Apophenia will treat your\nsettings group like any other, and users can use \\ref Apop_settings_add_group to\npopulate it and attach it to any model.\n\n\\section vtables Registering new methods in vtables\n\nThe settings groups are for adding arbitrary model-specific nouns; vtables are for\nadding arbitrary model-specific verbs.\n\nMany functions (e.g., entropy, the dlog likelihood, Bayesian updating) have\nspecial cases for well-known models like the Normal distribution. \nAny function may maintain a registry of models and associated special-case procedures, aka a vtable.\n\nLookups happen based on a hash that takes into account the elements of the model\nthat will be used in the calculation. For example, the \\c apop_update_hash takes in two\nmodels and calculates the hash based on the address of the prior's \\c draw method and\nthe likelihood's \\c log_likelihood or \\c p method. Thus, a vtable lookup for new models\nthat re-use the same methods (at the same addresses in memory) will still find the\nsame special-case function.\n\nIf you need to deregister the function, use the associated deregister function,\ne.g. apop_update_vtable_drop(apop_beta, apop_binomial). You can guarantee\nthat a method will not be re-added by following up the _drop with, e.g.,\napop_update_vtable_add(NULL, apop_beta, apop_binomial).\n\nThe steps for adding a function to an existing vtable:\n\n\\li See \\ref apop_update, \\ref apop_score, \\ref apop_predict, \\ref apop_model_print, and \\ref\napop_parameter_model for examples and procedure-specific details.\n\\li Write a function following the given type definition, as listed in the function's documentation.\n\\li Use the associated _vtable_add function to add the function and associate it\nwith the given model. For example, to add a Beta-binomial routine named \\c betabinom\nto the registry of Bayesian updating routines, use apop_update_vtable_add(betabinom,\napop_beta, apop_binomial).\n\\li Place a call to ..._vtable_add in the \\c prep method of the given model, thus ensuring that the auxiliary functions are registered after the first time the model is sent to \\ref apop_estimate.\n\nThe easiest way to set up a new vtable is to copy/paste/modify an existing one. Briefly:\n\n\\li See the existing setups in the vtables portion of apop.h. \n\\li Cut/paste one and do a search and replace to change the name to match your desired use.\n\\li Set the typedef to describe the functions that get added to the vtable.\n\\li Rewrite the hash function to check the part of the inputs that interest you. For\nexample, the update vtable associates functions with the \\c draw, \\c log_likelihood,\nand \\p methods of the model. A model where these elements are identical \nwill still match even if other elements are different.\n\n\\section modeldataparts The data elements\n\nThe remainder of this section covers the detailed expectations regarding the elements\nof the \\ref apop_model structure. I begin with the data (non-function) elements,\nand then cover the method (function) elements. Some of the following will be\nrequirements for all models and some will be advice to authors; I use the accepted\ndefinitions of \"must\", \"shall\", \"may\"\nand related words.\n\n\\subsection datasubsec Data\n\n\\li Each row of the \\c data element is treated as a single observation by many functions. \nFor example, \\ref apop_bootstrap_cov depends on each row being an iid observation to function correctly.\nCalculating the Bayesian Information Criterion (BIC) requires knowing\nthe number of observations in the data, and assumes that row count==observation count.\nFor complex data, the \\ref apop_data_pack and \\ref apop_data_unpack functions can help with this. \n\n\\li Some functions (bootstrap again, or many uses of \\ref apop_kl_divergence) use \\ref\napop_draw to use your model's RNG (or a default) to draw a\n value, write it to the matrix element of the data set, and then move on to an\n estimation or other step. In this case, the data sent in will be entirely in the \\c\n ->matrix element of the \\ref apop_data set sent to model methods. Your \\c likelihood, \\c p, \\c cdf, and \\c estimate routines\n must accept data as a single row of the matrix of the \\ref apop_data set for such functions to work.\n They may accept other formats. Tip: you can use \\ref apop_data_pack and \\ref apop_data_unpack to convert a structured set to a single row and back again.\n\n\\li Your routines may accept other data formats, as per contract with the user.\n For example, regression-type functions use a function named \\c ols_shuffle\n to convert a matrix where the first column is the dependent variable to a data\n set with dependent variable in the vector and a column of ones in the first\n matrix column.\n\n\\subsection paramsubsec Parameters, vsize, msize1, msize2\n\n\\li The sizes will be used by the \\c prep method of the model; see below. Given the model \\c m and its elements \\c m.vsize, \\c m.msize1, \\c m.msize2,\n functions that need to allocate a parameter set will do so via apop_data_alloc(m.vsize, m.msize1, m.msize2). \n\n\n\\subsection infosubsec Info\n\n\\li The first page, which should be named \\c <info>, is typically a list of scalars. Nothing is guaranteed, but the elements may include:\n\n\\li AIC: Aikake Information Criterion\n\\li AIC_c: AIC with a finite sample correction. ``Generally, we advocate the use of AIC_c when the ratio \\f$n/K\\f$ is small (say \\f$<\\f$ 40)'' [Kenneth P. Burnham, David R. Anderson: Model Selection and Multi-Model Inference, p 66, emphasis in original.]\n\\li BIC: Bayesian Information Criterion\n\\li R squared\n\\li R squared adj\n\\li log likelihood\n\\li status [0=OK, nozero=other].\n\nFor those elements that require a count of input data, the calculations assume each row in the input \\ref apop_data set is a single datum.\n\nGet these via, e.g., apop_data_get(your_model->info, .rowname=\"log likelihood\").\nWhen writing for any arbitrary function, be prepared to handle \\c NaN, indicating that the element is not calculated or saved in the info page by the given model.\n\nFor OLS-type estimations, each row corresponds to the row in the original data. For\nfilling in of missing data, the elements may appear anywhere, so the row/col indices are\nessential.\n\n\\subsection settingsgroupmention settings, more\n\nIn object-oriented jargon, settings groups are the private elements of the data set,\nto be pulled out in certain contexts, and ignored in all others. Therefore, there are\nno rules about internal use. The \\c more element of the \\ref apop_model provides a lightweight\nmeans of attaching an arbitrary struct to a model. See \\ref settingswriting above for details.\n\n\\li As many settings groups of different types as desired can be added to a single \\ref apop_model.\n\\li One \\ref apop_model can not hold two settings groups of the same type. Re-additions cause the removal of the previous version of the group.\n\\li If the \\c more pointer points to a structure or value (let it be \\c ss), then \\c more_size must be set to sizeof(ss).\n\n\n\\section methodsection Methods\n\n\\subsection psubsection p, log_likelihood\n\n\\li Function headers look like long double your_p_or_ll(apop_data *d, apop_model *params).\n\\li The inputs are an \\ref apop_data set and an \\ref apop_model, which should include the elements needed to fully estimate the probability/likelihood (probably a filled ->parameters element, possibly a settings group added by the user).\n\\li Assume that the parameters have been set, by users via \\ref apop_estimate or \\ref apop_model_set_parameters, or by \\ref apop_maximum_likelihood by its search algorithms. If the parameters are necessary, the function shall check that the parameters are not \\c NULL and set the model's \\c error element to \\c 'p' if they are missing.\n\\li Return \\c NaN on errors. If an error in the input model is found, the function may set the input model's \\c error element to an appropriate \\c char value.\n\\li If your model includes both \\c log_likelihood and \\c p methods, it must be the case that log(p(d, m)) equals log_likelihood(d, m) for all \\c d and \\c m. This implies that \\c p must return a value \\f$\\geq 0\\f$. Note that \\ref apop_maximum_likelihood will accept functions where \\c p returns a negative value, but diagonstics that depend on log likelihood like AIC will return NaN.\n\\li If observations are assumed to be iid, you may be able to use \\ref apop_map_sum to write the core of the log likelihood function.\n\n\\subsection prepsubsection prep\n\n\\li Function header looks like void your_prep(apop_data *data, apop_model *params).\n\\li Re-prepping a model after it has already been prepped shall have no effect. Where there is ambiguity with the other requirements, this takes precedence.\n\\li The model's data pointer shall be set to point to the input data.\n\\li The \\c info element shall be allocated and its title set to \\.\n\\li If \\c vsize, \\c msize1, or \\c msize2 are -1, then the prep function shall set them to the width of the input data.\n\\li If \\c dsize is -1, then the prep function shall set it to the width of the input data.\n\\li If the \\c parameters element is not allocated, the function shall allocate it via apop_data_alloc(vsize, msize1, msize2) (or equivalent).\n\\li The default is \\ref apop_model_clear. It does all of the above.\n\\li The input data may be modified by the prep routine. For example, the \\ref apop_ols prep routine shuffles a single input matrix as described above under \\c data, and the \\ref apop_pmf prep routine calls \\ref apop_data_pmf_compress on the input data.\n\\li The prep routine may initialize any desired settings groups. Unless otherwise\nstated, these should not be removed if they are already there, so that users can override defaults by adding a settings group before starting an estimation.\n\\li If any functions associated with the model need to be added to \na vtable (see above), the registration shall happen here. Registration may also happen elsewhere.\n\n\\subsection estimatesubsection estimate\n\n\\li Function header looks like void your_estimate(apop_data *data, apop_model *params).\nIt modifies the input model, and returns nothing. Note that this is different from the wrapper function, \\ref apop_estimate, which makes a copy of its input model, preps it, and then calls the \\c estimate function with the prepeped copy.\n\\li Assume that the prep routine has already been run. Notably, this means that parameters have been allocated.\n\\li Assume that the \\c parameters hold garbage (as in a \\c malloc without a subsequent assignment to the malloc-ed space).\n\\li The function shall set the \\c parameters of the input model. For consistency with other models, the estimate should be the maximum likelihood estimate, unless otherwise documented.\n\\li Additional settings may be set.\n\\li The model's \\c <Info> page may be filled with statistics, as discussed at infosubsec. For scalars like log likelihood and AIC, use \\ref apop_data_add_named_elmt.\n\\li Data should not be modified by the \\c estimate routine; any changes to the data made by \\c estimate must be documented.\n\\li The default called by \\ref apop_estimate is \\ref apop_maximum_likelihood.\n\\li If errors occur during processing, set the model's \\c error element to a single character. Documentation should include the list of error characters and their meaning.\n\n\\subsection drawsubsection draw\n\n\\li Function header looks like void your_draw(double *out, gsl_rng* r, apop_model *params)\n\\li Assume that model \\c paramters are set, via \\ref apop_estimate or \\ref apop_model_set_parameters. The author of the draw method should check that \\c parameters are not \\c NULL if needed and fill the output with NaNs if necessary parameters are not set.\n\\li Caller inputs a pointer-to-double of length \\c dsize; user is expected to make sure that there is adequate space. Caller also inputs a \\c gsl_rng, already allocated (probably via \\ref apop_rng_alloc, possibly from \\ref apop_rng_get_thread).\n\\li The function shall fill the space pointed to by the input pointer with a random draw from the data space, where the likelihood of any given observation is proportional to its likelihood as given by the \\c p method. Data shall be reduced to a single vector via \\ref apop_data_pack if it is not already a single vector.\n\n\\subsection cdfsubsection cdf\n\n\\li Function header looks like long double your_cdf(apop_data *d, apop_model *params).\n\\li Assume that \\c parameters are set, via \\ref apop_estimate or \\ref apop_model_set_parameters. The author of the CDF method should check that \\c parameters are not \\c NULL and return NaN if necessary parameters are not set.\n\\li The CDF method must accept data as a single row of data in the \\c matrix of the input \\ref apop_data set (as per a draw produced using the \\c draw method). May accept other formats.\n\\li Returns the percentage of the likelihood function \\f$\\leq\\f$ the first row of the input data. The definition of \\f$\\leq\\f$ is chosen by the model author.\n\\li If one is not already present, an \\c apop_cdf_settings group may be added to the model to store temp data. See the \\ref apop_cdf function for details.\n\n\\subsection constraintsubsection constraint\n\n\\li Function header looks like long double your_constraint(apop_data *data, apop_model *params).\n\\li Assume that \\c parameters are set, via \\ref apop_estimate, \\ref apop_model_set_parameters, or the internals of an MLE search. The author of the constraint method should check that \\c parameters are not \\c NULL and return NaN if necessary parameters are not set.\n\\li See \\ref apop_linear_constraint for a useful basis and/or example. Many constraints can be written as wrappers for this function.\n\\li If the constraint is met, then return zero.\n\\li If the constraint fails, then (1) move the \\c parameters in the input model to a\nconstraint-satisfying value, and (2) return the distance between the input parameters and\nwhat you've moved the parameters to. The choice of within-bounds parameters and distance function is left to the author of the constraint function.\n*/\n\n\n/**\\defgroup models \n\nThis section is a detailed description of the stock models that ship with Apophenia.\nIt is a reference. For an explanation of what to do with an \\ref apop_model, see \\ref modelsec.\n\nThe primary questions one has about a model in practice are what format the input data should take and what to expect of an estimated output.\n\nGenerally, the input data consists of an \\ref apop_data set where each row is a single observation. Details beyond that are listed below.\n\nThe output after running \\ref apop_estimate to produce a fitted model are generally\nfound in three places: the vector of the output parameter set, its matrix, or a new\nsettings group. The basic intuition is that \nif the parameters are always a short list of scalars, they are in the vector;\nif there exists a situation where they could take matrix form, the parameters will be in the matrix;\nif they require more structure than that, they will be a settings group.\n\nIf the basic structure of the \\ref apop_data set is unfamiliar to you, see \\ref\ndataoverview, which will discuss the basic means of getting data out of a struct. For\nexample, the estimated \\ref apop_normal distribution has the mean in position zero of\nthe vector and the standard deviation in position one, so they could be extracted as follows:\n\n\\code\napop_data *d = apop_text_to_data(\"sample data from before\")\napop_model *out = apop_estimate(d, apop_normal);\ndouble mu = apop_data_get(out>parameters, 0);\ndouble sigma = apop_data_get(out>parameters, 1);\n\n//What is the p-value of test whose null hypothesis is that μ=3.3?\nprintf (\"pval=%g\\n\", apop_test(3.3, \"normal\", mu, sigma);\n\\endcode\n\nSee \\ref modelsec for discussion of how to pull settings groups using \\ref\nApop_settings_get (for one item) or \\ref apop_settings_get_group (for a full settings\ngroup).\n*/\n" }, { "path": "docs/doxygen.conf.in", "content": "# Doxyfile 1.8.9.1\n\n# This file describes the settings to be used by the documentation system\n# doxygen (www.doxygen.org) for a project.\n#\n# All text after a double hash (##) is considered a comment and is placed in\n# front of the TAG it is preceding.\n#\n# All text after a single hash (#) is considered a comment and will be ignored.\n# The format is:\n# TAG = value [value, ...]\n# For lists, items can also be appended using:\n# TAG += value [value, ...]\n# Values that contain spaces should be placed between quotes (\\\" \\\").\n\n#---------------------------------------------------------------------------\n# Project related configuration options\n#---------------------------------------------------------------------------\n\n# This tag specifies the encoding used for all characters in the config file\n# that follow. The default is UTF-8 which is also the encoding used for all text\n# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv\n# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv\n# for the list of possible encodings.\n# The default value is: UTF-8.\n\nDOXYFILE_ENCODING = UTF-8\n\n# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by\n# double-quotes, unless you are using Doxywizard) that should identify the\n# project for which the documentation is generated. This name is used in the\n# title of most generated pages and in a few other places.\n# The default value is: My Project.\n\nPROJECT_NAME = Apophenia\n\n# The PROJECT_NUMBER tag can be used to enter a project or revision number. This\n# could be handy for archiving the generated documentation or if some version\n# control system is used.\n\nPROJECT_NUMBER =\n\n# Using the PROJECT_BRIEF tag one can provide an optional one line description\n# for a project that appears at the top of each page and should give viewer a\n# quick idea about the purpose of the project. Keep the description short.\n\nPROJECT_BRIEF =\n\n# With the PROJECT_LOGO tag one can specify a logo or an icon that is included\n# in the documentation. The maximum height of the logo should not exceed 55\n# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy\n# the logo to the output directory.\n\nPROJECT_LOGO =\n\n# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path\n# into which the generated documentation will be written. If a relative path is\n# entered, it will be relative to the location where doxygen was started. If\n# left blank the current directory will be used.\n\nOUTPUT_DIRECTORY =\n\n# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub-\n# directories (in 2 levels) under the output directory of each output format and\n# will distribute the generated files over these directories. Enabling this\n# option can be useful when feeding doxygen a huge amount of source files, where\n# putting all generated files in the same directory would otherwise causes\n# performance problems for the file system.\n# The default value is: NO.\n\nCREATE_SUBDIRS = NO\n\n# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII\n# characters to appear in the names of generated files. If set to NO, non-ASCII\n# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode\n# U+3044.\n# The default value is: NO.\n\nALLOW_UNICODE_NAMES = YES\n\n# The OUTPUT_LANGUAGE tag is used to specify the language in which all\n# documentation generated by doxygen is written. Doxygen will use this\n# information to generate all constant output in the proper language.\n# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese,\n# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States),\n# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian,\n# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages),\n# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian,\n# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian,\n# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish,\n# Ukrainian and Vietnamese.\n# The default value is: English.\n\nOUTPUT_LANGUAGE = English\n\n# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member\n# descriptions after the members that are listed in the file and class\n# documentation (similar to Javadoc). Set to NO to disable this.\n# The default value is: YES.\n\nBRIEF_MEMBER_DESC = YES\n\n# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief\n# description of a member or function before the detailed description\n#\n# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the\n# brief descriptions will be completely suppressed.\n# The default value is: YES.\n\nREPEAT_BRIEF = YES\n\n# This tag implements a quasi-intelligent brief description abbreviator that is\n# used to form the text in various listings. Each string in this list, if found\n# as the leading text of the brief description, will be stripped from the text\n# and the result, after processing the whole list, is used as the annotated\n# text. Otherwise, the brief description is used as-is. If left blank, the\n# following values are used ($name is automatically replaced with the name of\n# the entity):The $name class, The $name widget, The $name file, is, provides,\n# specifies, contains, represents, a, an and the.\n\nABBREVIATE_BRIEF =\n\n# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then\n# doxygen will generate a detailed section even if there is only a brief\n# description.\n# The default value is: NO.\n\nALWAYS_DETAILED_SEC = NO\n\n# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all\n# inherited members of a class in the documentation of that class as if those\n# members were ordinary class members. Constructors, destructors and assignment\n# operators of the base classes will not be shown.\n# The default value is: NO.\n\nINLINE_INHERITED_MEMB = NO\n\n# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path\n# before files name in the file list and in the header files. If set to NO the\n# shortest path that makes the file name unique will be used\n# The default value is: YES.\n\nFULL_PATH_NAMES = NO\n\n# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path.\n# Stripping is only done if one of the specified strings matches the left-hand\n# part of the path. The tag can be used to show relative paths in the file list.\n# If left blank the directory from which doxygen is run is used as the path to\n# strip.\n#\n# Note that you can specify absolute paths here, but also relative paths, which\n# will be relative from the directory where doxygen is started.\n# This tag requires that the tag FULL_PATH_NAMES is set to YES.\n\nSTRIP_FROM_PATH =\n\n# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the\n# path mentioned in the documentation of a class, which tells the reader which\n# header file to include in order to use a class. If left blank only the name of\n# the header file containing the class definition is used. Otherwise one should\n# specify the list of include paths that are normally passed to the compiler\n# using the -I flag.\n\nSTRIP_FROM_INC_PATH =\n\n# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but\n# less readable) file names. This can be useful is your file systems doesn't\n# support long names like on DOS, Mac, or CD-ROM.\n# The default value is: NO.\n\nSHORT_NAMES = NO\n\n# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the\n# first line (until the first dot) of a Javadoc-style comment as the brief\n# description. If set to NO, the Javadoc-style will behave just like regular Qt-\n# style comments (thus requiring an explicit @brief command for a brief\n# description.)\n# The default value is: NO.\n\nJAVADOC_AUTOBRIEF = NO\n\n# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first\n# line (until the first dot) of a Qt-style comment as the brief description. If\n# set to NO, the Qt-style will behave just like regular Qt-style comments (thus\n# requiring an explicit \\brief command for a brief description.)\n# The default value is: NO.\n\nQT_AUTOBRIEF = NO\n\n# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a\n# multi-line C++ special comment block (i.e. a block of //! or /// comments) as\n# a brief description. This used to be the default behavior. The new default is\n# to treat a multi-line C++ comment block as a detailed description. Set this\n# tag to YES if you prefer the old behavior instead.\n#\n# Note that setting this tag to YES also means that rational rose comments are\n# not recognized any more.\n# The default value is: NO.\n\nMULTILINE_CPP_IS_BRIEF = NO\n\n# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the\n# documentation from any documented member that it re-implements.\n# The default value is: YES.\n\nINHERIT_DOCS = YES\n\n# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new\n# page for each member. If set to NO, the documentation of a member will be part\n# of the file/class/namespace that contains it.\n# The default value is: NO.\n\nSEPARATE_MEMBER_PAGES = NO\n\n# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen\n# uses this value to replace tabs by spaces in code fragments.\n# Minimum value: 1, maximum value: 16, default value: 4.\n\nTAB_SIZE = 5\n\n# This tag can be used to specify a number of aliases that act as commands in\n# the documentation. An alias has the form:\n# name=value\n# For example adding\n# \"sideeffect=@par Side Effects:\\n\"\n# will allow you to put the command \\sideeffect (or @sideeffect) in the\n# documentation, which will result in a user-defined paragraph with heading\n# \"Side Effects:\". You can put \\n's in the value part of an alias to insert\n# newlines.\n\nALIASES =\n\n# This tag can be used to specify a number of word-keyword mappings (TCL only).\n# A mapping has the form \"name=value\". For example adding \"class=itcl::class\"\n# will allow you to use the command class in the itcl::class meaning.\n\nTCL_SUBST =\n\n# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources\n# only. Doxygen will then generate output that is more tailored for C. For\n# instance, some of the names that are used will be different. The list of all\n# members will be omitted, etc.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_FOR_C = YES\n\n# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or\n# Python sources only. Doxygen will then generate output that is more tailored\n# for that language. For instance, namespaces will be presented as packages,\n# qualified scopes will look different, etc.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_JAVA = NO\n\n# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran\n# sources. Doxygen will then generate output that is tailored for Fortran.\n# The default value is: NO.\n\nOPTIMIZE_FOR_FORTRAN = NO\n\n# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL\n# sources. Doxygen will then generate output that is tailored for VHDL.\n# The default value is: NO.\n\nOPTIMIZE_OUTPUT_VHDL = NO\n\n# Doxygen selects the parser to use depending on the extension of the files it\n# parses. With this tag you can assign which parser to use for a given\n# extension. Doxygen has a built-in mapping, but you can override or extend it\n# using this tag. The format is ext=language, where ext is a file extension, and\n# language is one of the parsers supported by doxygen: IDL, Java, Javascript,\n# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran:\n# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran:\n# Fortran. In the later case the parser tries to guess whether the code is fixed\n# or free formatted code, this is the default for Fortran type files), VHDL. For\n# instance to make doxygen treat .inc files as Fortran files (default is PHP),\n# and .f files as C (default is Fortran), use: inc=Fortran f=C.\n#\n# Note: For files without extension you can use no_extension as a placeholder.\n#\n# Note that for custom extensions you also need to set FILE_PATTERNS otherwise\n# the files are not read by doxygen.\n\nEXTENSION_MAPPING =\n\n# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments\n# according to the Markdown format, which allows for more readable\n# documentation. See http://daringfireball.net/projects/markdown/ for details.\n# The output of markdown processing is further processed by doxygen, so you can\n# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in\n# case of backward compatibilities issues.\n# The default value is: YES.\n\nMARKDOWN_SUPPORT = YES\n\n# When enabled doxygen tries to link words that correspond to documented\n# classes, or namespaces to their corresponding documentation. Such a link can\n# be prevented in individual cases by putting a % sign in front of the word or\n# globally by setting AUTOLINK_SUPPORT to NO.\n# The default value is: YES.\n\nAUTOLINK_SUPPORT = YES\n\n# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want\n# to include (a tag file for) the STL sources as input, then you should set this\n# tag to YES in order to let doxygen match functions declarations and\n# definitions whose arguments contain STL classes (e.g. func(std::string);\n# versus func(std::string) {}). This also make the inheritance and collaboration\n# diagrams that involve STL classes more complete and accurate.\n# The default value is: NO.\n\nBUILTIN_STL_SUPPORT = NO\n\n# If you use Microsoft's C++/CLI language, you should set this option to YES to\n# enable parsing support.\n# The default value is: NO.\n\nCPP_CLI_SUPPORT = NO\n\n# Set the SIP_SUPPORT tag to YES if your project consists of sip (see:\n# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen\n# will parse them like normal C++ but will assume all classes use public instead\n# of private inheritance when no explicit protection keyword is present.\n# The default value is: NO.\n\nSIP_SUPPORT = NO\n\n# For Microsoft's IDL there are propget and propput attributes to indicate\n# getter and setter methods for a property. Setting this option to YES will make\n# doxygen to replace the get and set methods by a property in the documentation.\n# This will only work if the methods are indeed getting or setting a simple\n# type. If this is not the case, or you want to show the methods anyway, you\n# should set this option to NO.\n# The default value is: YES.\n\nIDL_PROPERTY_SUPPORT = YES\n\n# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC\n# tag is set to YES then doxygen will reuse the documentation of the first\n# member in the group (if any) for the other members of the group. By default\n# all members of a group must be documented explicitly.\n# The default value is: NO.\n\nDISTRIBUTE_GROUP_DOC = NO\n\n# Set the SUBGROUPING tag to YES to allow class member groups of the same type\n# (for instance a group of public functions) to be put as a subgroup of that\n# type (e.g. under the Public Functions section). Set it to NO to prevent\n# subgrouping. Alternatively, this can be done per class using the\n# \\nosubgrouping command.\n# The default value is: YES.\n\nSUBGROUPING = YES\n\n# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions\n# are shown inside the group in which they are included (e.g. using \\ingroup)\n# instead of on a separate page (for HTML and Man pages) or section (for LaTeX\n# and RTF).\n#\n# Note that this feature does not work in combination with\n# SEPARATE_MEMBER_PAGES.\n# The default value is: NO.\n\nINLINE_GROUPED_CLASSES = NO\n\n# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions\n# with only public data fields or simple typedef fields will be shown inline in\n# the documentation of the scope in which they are defined (i.e. file,\n# namespace, or group documentation), provided this scope is documented. If set\n# to NO, structs, classes, and unions are shown on a separate page (for HTML and\n# Man pages) or section (for LaTeX and RTF).\n# The default value is: NO.\n\nINLINE_SIMPLE_STRUCTS = NO\n\n# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or\n# enum is documented as struct, union, or enum with the name of the typedef. So\n# typedef struct TypeS {} TypeT, will appear in the documentation as a struct\n# with name TypeT. When disabled the typedef will appear as a member of a file,\n# namespace, or class. And the struct will be named TypeS. This can typically be\n# useful for C code in case the coding convention dictates that all compound\n# types are typedef'ed and only the typedef is referenced, never the tag name.\n# The default value is: NO.\n\nTYPEDEF_HIDES_STRUCT = YEs\n\n# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This\n# cache is used to resolve symbols given their name and scope. Since this can be\n# an expensive process and often the same symbol appears multiple times in the\n# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small\n# doxygen will become slower. If the cache is too large, memory is wasted. The\n# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range\n# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536\n# symbols. At the end of a run doxygen will report the cache usage and suggest\n# the optimal cache size from a speed point of view.\n# Minimum value: 0, maximum value: 9, default value: 0.\n\nLOOKUP_CACHE_SIZE = 0\n\n#---------------------------------------------------------------------------\n# Build related configuration options\n#---------------------------------------------------------------------------\n\n# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in\n# documentation are documented, even if no documentation was available. Private\n# class members and static file members will be hidden unless the\n# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES.\n# Note: This will also disable the warnings about undocumented members that are\n# normally produced when WARNINGS is set to YES.\n# The default value is: NO.\n\nEXTRACT_ALL = NO\n\n# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will\n# be included in the documentation.\n# The default value is: NO.\n\nEXTRACT_PRIVATE = NO\n\n# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal\n# scope will be included in the documentation.\n# The default value is: NO.\n\nEXTRACT_PACKAGE = NO\n\n# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be\n# included in the documentation.\n# The default value is: NO.\n\nEXTRACT_STATIC = NO\n\n# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined\n# locally in source files will be included in the documentation. If set to NO,\n# only classes defined in header files are included. Does not have any effect\n# for Java sources.\n# The default value is: YES.\n\nEXTRACT_LOCAL_CLASSES = YES\n\n# This flag is only useful for Objective-C code. If set to YES, local methods,\n# which are defined in the implementation section but not in the interface are\n# included in the documentation. If set to NO, only methods in the interface are\n# included.\n# The default value is: NO.\n\nEXTRACT_LOCAL_METHODS = NO\n\n# If this flag is set to YES, the members of anonymous namespaces will be\n# extracted and appear in the documentation as a namespace called\n# 'anonymous_namespace{file}', where file will be replaced with the base name of\n# the file that contains the anonymous namespace. By default anonymous namespace\n# are hidden.\n# The default value is: NO.\n\nEXTRACT_ANON_NSPACES = NO\n\n# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all\n# undocumented members inside documented classes or files. If set to NO these\n# members will be included in the various overviews, but no documentation\n# section is generated. This option has no effect if EXTRACT_ALL is enabled.\n# The default value is: NO.\n\nHIDE_UNDOC_MEMBERS = NO\n\n# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all\n# undocumented classes that are normally visible in the class hierarchy. If set\n# to NO, these classes will be included in the various overviews. This option\n# has no effect if EXTRACT_ALL is enabled.\n# The default value is: NO.\n\nHIDE_UNDOC_CLASSES = NO\n\n# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend\n# (class|struct|union) declarations. If set to NO, these declarations will be\n# included in the documentation.\n# The default value is: NO.\n\nHIDE_FRIEND_COMPOUNDS = NO\n\n# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any\n# documentation blocks found inside the body of a function. If set to NO, these\n# blocks will be appended to the function's detailed documentation block.\n# The default value is: NO.\n\nHIDE_IN_BODY_DOCS = NO\n\n# The INTERNAL_DOCS tag determines if documentation that is typed after a\n# \\internal command is included. If the tag is set to NO then the documentation\n# will be excluded. Set it to YES to include the internal documentation.\n# The default value is: NO.\n\nINTERNAL_DOCS = NO\n\n# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file\n# names in lower-case letters. If set to YES, upper-case letters are also\n# allowed. This is useful if you have classes or files whose names only differ\n# in case and if your file system supports case sensitive file names. Windows\n# and Mac users are advised to set this option to NO.\n# The default value is: system dependent.\n\nCASE_SENSE_NAMES = YES\n\n# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with\n# their full class and namespace scopes in the documentation. If set to YES, the\n# scope will be hidden.\n# The default value is: NO.\n\nHIDE_SCOPE_NAMES = NO\n\n# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will\n# append additional text to a page's title, such as Class Reference. If set to\n# YES the compound reference will be hidden.\n# The default value is: NO.\n\nHIDE_COMPOUND_REFERENCE= NO\n\n# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of\n# the files that are included by a file in the documentation of that file.\n# The default value is: YES.\n\nSHOW_INCLUDE_FILES = NO\n\n# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each\n# grouped member an include statement to the documentation, telling the reader\n# which file to include in order to use the member.\n# The default value is: NO.\n\nSHOW_GROUPED_MEMB_INC = NO\n\n# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include\n# files with double quotes in the documentation rather than with sharp brackets.\n# The default value is: NO.\n\nFORCE_LOCAL_INCLUDES = NO\n\n# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the\n# documentation for inline members.\n# The default value is: YES.\n\nINLINE_INFO = YES\n\n# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the\n# (detailed) documentation of file and class members alphabetically by member\n# name. If set to NO, the members will appear in declaration order.\n# The default value is: YES.\n\nSORT_MEMBER_DOCS = YES\n\n# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief\n# descriptions of file, namespace and class members alphabetically by member\n# name. If set to NO, the members will appear in declaration order. Note that\n# this will also influence the order of the classes in the class list.\n# The default value is: NO.\n\nSORT_BRIEF_DOCS = YES\n\n# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the\n# (brief and detailed) documentation of class members so that constructors and\n# destructors are listed first. If set to NO the constructors will appear in the\n# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS.\n# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief\n# member documentation.\n# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting\n# detailed member documentation.\n# The default value is: NO.\n\nSORT_MEMBERS_CTORS_1ST = NO\n\n# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy\n# of group names into alphabetical order. If set to NO the group names will\n# appear in their defined order.\n# The default value is: NO.\n\nSORT_GROUP_NAMES = NO\n\n# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by\n# fully-qualified names, including namespaces. If set to NO, the class list will\n# be sorted only by class name, not including the namespace part.\n# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES.\n# Note: This option applies only to the class list, not to the alphabetical\n# list.\n# The default value is: NO.\n\nSORT_BY_SCOPE_NAME = NO\n\n# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper\n# type resolution of all parameters of a function it will reject a match between\n# the prototype and the implementation of a member function even if there is\n# only one candidate or it is obvious which candidate to choose by doing a\n# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still\n# accept a match between prototype and implementation in such cases.\n# The default value is: NO.\n\nSTRICT_PROTO_MATCHING = NO\n\n# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo\n# list. This list is created by putting \\todo commands in the documentation.\n# The default value is: YES.\n\nGENERATE_TODOLIST = NO\n\n# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test\n# list. This list is created by putting \\test commands in the documentation.\n# The default value is: YES.\n\nGENERATE_TESTLIST = YES\n\n# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug\n# list. This list is created by putting \\bug commands in the documentation.\n# The default value is: YES.\n\nGENERATE_BUGLIST = YES\n\n# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO)\n# the deprecated list. This list is created by putting \\deprecated commands in\n# the documentation.\n# The default value is: YES.\n\nGENERATE_DEPRECATEDLIST= NO\n\n# The ENABLED_SECTIONS tag can be used to enable conditional documentation\n# sections, marked by \\if ... \\endif and \\cond \n# ... \\endcond blocks.\n\nENABLED_SECTIONS =\n\n# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the\n# initial value of a variable or macro / define can have for it to appear in the\n# documentation. If the initializer consists of more lines than specified here\n# it will be hidden. Use a value of 0 to hide initializers completely. The\n# appearance of the value of individual variables and macros / defines can be\n# controlled using \\showinitializer or \\hideinitializer command in the\n# documentation regardless of this setting.\n# Minimum value: 0, maximum value: 10000, default value: 30.\n\nMAX_INITIALIZER_LINES = 0\n\n# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at\n# the bottom of the documentation of classes and structs. If set to YES, the\n# list will mention the files that were used to generate the documentation.\n# The default value is: YES.\n\nSHOW_USED_FILES = NO\n\n# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This\n# will remove the Files entry from the Quick Index and from the Folder Tree View\n# (if specified).\n# The default value is: YES.\n\nSHOW_FILES = NO\n\n# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces\n# page. This will remove the Namespaces entry from the Quick Index and from the\n# Folder Tree View (if specified).\n# The default value is: YES.\n\nSHOW_NAMESPACES = NO\n\n# The FILE_VERSION_FILTER tag can be used to specify a program or script that\n# doxygen should invoke to get the current version for each file (typically from\n# the version control system). Doxygen will invoke the program by executing (via\n# popen()) the command command input-file, where command is the value of the\n# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided\n# by doxygen. Whatever the program writes to standard output is used as the file\n# version. For an example see the documentation.\n\nFILE_VERSION_FILTER =\n\n# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed\n# by doxygen. The layout file controls the global structure of the generated\n# output files in an output format independent way. To create the layout file\n# that represents doxygen's defaults, run doxygen with the -l option. You can\n# optionally specify a file name after the option, if omitted DoxygenLayout.xml\n# will be used as the name of the layout file.\n#\n# Note that if you run doxygen from a directory containing a file called\n# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE\n# tag is left empty.\n\nLAYOUT_FILE =\n\n# The CITE_BIB_FILES tag can be used to specify one or more bib files containing\n# the reference definitions. This must be a list of .bib files. The .bib\n# extension is automatically appended if omitted. This requires the bibtex tool\n# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info.\n# For LaTeX the style of the bibliography can be controlled using\n# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the\n# search path. See also \\cite for info how to create references.\n\nCITE_BIB_FILES =\n\n#---------------------------------------------------------------------------\n# Configuration options related to warning and progress messages\n#---------------------------------------------------------------------------\n\n# The QUIET tag can be used to turn on/off the messages that are generated to\n# standard output by doxygen. If QUIET is set to YES this implies that the\n# messages are off.\n# The default value is: NO.\n\nQUIET = YES\n\n# The WARNINGS tag can be used to turn on/off the warning messages that are\n# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES\n# this implies that the warnings are on.\n#\n# Tip: Turn warnings on while writing the documentation.\n# The default value is: YES.\n\nWARNINGS = YES\n\n# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate\n# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag\n# will automatically be disabled.\n# The default value is: YES.\n\nWARN_IF_UNDOCUMENTED = NO\n\n# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for\n# potential errors in the documentation, such as not documenting some parameters\n# in a documented function, or documenting parameters that don't exist or using\n# markup commands wrongly.\n# The default value is: YES.\n\nWARN_IF_DOC_ERROR = YES\n\n# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that\n# are documented, but have no documentation for their parameters or return\n# value. If set to NO, doxygen will only warn about wrong or incomplete\n# parameter documentation, but not about the absence of documentation.\n# The default value is: NO.\n\nWARN_NO_PARAMDOC = NO\n\n# The WARN_FORMAT tag determines the format of the warning messages that doxygen\n# can produce. The string should contain the $file, $line, and $text tags, which\n# will be replaced by the file and line number from which the warning originated\n# and the warning text. Optionally the format may contain $version, which will\n# be replaced by the version of the file (if it could be obtained via\n# FILE_VERSION_FILTER)\n# The default value is: $file:$line: $text.\n\nWARN_FORMAT = \"$file:$line: $text\"\n\n# The WARN_LOGFILE tag can be used to specify a file to which warning and error\n# messages should be written. If left blank the output is written to standard\n# error (stderr).\n\nWARN_LOGFILE = doxygen.log\n\n#---------------------------------------------------------------------------\n# Configuration options related to the input files\n#---------------------------------------------------------------------------\n\n# The INPUT tag is used to specify the files and/or directories that contain\n# documented source files. You may enter file names like myfile.cpp or\n# directories like /usr/src/myproject. Separate the files or directories with\n# spaces.\n# Note: If this tag is empty the current directory is searched.\n\nINPUT = @abs_top_builddir@/docs/include @abs_top_srcdir@\n\n# This tag can be used to specify the character encoding of the source files\n# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses\n# libiconv (or the iconv built into libc) for the transcoding. See the libiconv\n# documentation (see: http://www.gnu.org/software/libiconv) for the list of\n# possible encodings.\n# The default value is: UTF-8.\n\nINPUT_ENCODING = UTF-8\n\n# If the value of the INPUT tag contains directories, you can use the\n# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and\n# *.h) to filter out the source-files in the directories. If left blank the\n# following patterns are tested:*.c, *.cc, *.cxx, *.cpp, *.c++, *.java, *.ii,\n# *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, *.hh, *.hxx, *.hpp,\n# *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, *.m, *.markdown,\n# *.md, *.mm, *.dox, *.py, *.f90, *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf,\n# *.qsf, *.as and *.js.\n\nFILE_PATTERNS =\n\n# The RECURSIVE tag can be used to specify whether or not subdirectories should\n# be searched for input files as well.\n# The default value is: NO.\n\nRECURSIVE = YES\n\n# The EXCLUDE tag can be used to specify files and/or directories that should be\n# excluded from the INPUT source files. This way you can easily exclude a\n# subdirectory from a directory tree whose root is specified with the INPUT tag.\n#\n# Note that relative paths are relative to the directory from which doxygen is\n# run.\n\nEXCLUDE =\n\n# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or\n# directories that are symbolic links (a Unix file system feature) are excluded\n# from the input.\n# The default value is: NO.\n\nEXCLUDE_SYMLINKS = NO\n\n# If the value of the INPUT tag contains directories, you can use the\n# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude\n# certain files from those directories.\n#\n# Note that the wildcards are matched against the file with absolute path, so to\n# exclude all test directories for example use the pattern */test/*\n\nEXCLUDE_PATTERNS =\n\n# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names\n# (namespaces, classes, functions, etc.) that should be excluded from the\n# output. The symbol name can be a fully qualified name, a word, or if the\n# wildcard * is used, a substring. Examples: ANamespace, AClass,\n# AClass::ANamespace, ANamespace::*Test\n#\n# Note that the wildcards are matched against the file with absolute path, so to\n# exclude all test directories use the pattern */test/*\n\nEXCLUDE_SYMBOLS =\n\n# The EXAMPLE_PATH tag can be used to specify one or more files or directories\n# that contain example code fragments that are included (see the \\include\n# command).\n\nEXAMPLE_PATH = @abs_top_srcdir@\n\n# If the value of the EXAMPLE_PATH tag contains directories, you can use the\n# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and\n# *.h) to filter out the source-files in the directories. If left blank all\n# files are included.\n\nEXAMPLE_PATTERNS =\n\n# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be\n# searched for input files to be used with the \\include or \\dontinclude commands\n# irrespective of the value of the RECURSIVE tag.\n# The default value is: NO.\n\nEXAMPLE_RECURSIVE = YES\n\n# The IMAGE_PATH tag can be used to specify one or more files or directories\n# that contain images that are to be included in the documentation (see the\n# \\image command).\n\nIMAGE_PATH = .\n\n# The INPUT_FILTER tag can be used to specify a program that doxygen should\n# invoke to filter for each input file. Doxygen will invoke the filter program\n# by executing (via popen()) the command:\n#\n# \n#\n# where is the value of the INPUT_FILTER tag, and is the\n# name of an input file. Doxygen will then use the output that the filter\n# program writes to standard output. If FILTER_PATTERNS is specified, this tag\n# will be ignored.\n#\n# Note that the filter must not add or remove lines; it is applied before the\n# code is scanned, but not when the output code is generated. If lines are added\n# or removed, the anchors will not be placed correctly.\n\nINPUT_FILTER =\n\n# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern\n# basis. Doxygen will compare the file name with each pattern and apply the\n# filter if there is a match. The filters are a list of the form: pattern=filter\n# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how\n# filters are used. If the FILTER_PATTERNS tag is empty or if none of the\n# patterns match the file name, INPUT_FILTER is applied.\n\nFILTER_PATTERNS =\n\n# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using\n# INPUT_FILTER) will also be used to filter the input files that are used for\n# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES).\n# The default value is: NO.\n\nFILTER_SOURCE_FILES = NO\n\n# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file\n# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and\n# it is also possible to disable source filtering for a specific pattern using\n# *.ext= (so without naming a filter).\n# This tag requires that the tag FILTER_SOURCE_FILES is set to YES.\n\nFILTER_SOURCE_PATTERNS =\n\n# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that\n# is part of the input, its contents will be placed on the main page\n# (index.html). This can be useful if you have a project on for instance GitHub\n# and want to reuse the introduction page also for the doxygen output.\n\nUSE_MDFILE_AS_MAINPAGE =\n\n#---------------------------------------------------------------------------\n# Configuration options related to source browsing\n#---------------------------------------------------------------------------\n\n# If the SOURCE_BROWSER tag is set to YES then a list of source files will be\n# generated. Documented entities will be cross-referenced with these sources.\n#\n# Note: To get rid of all source code in the generated output, make sure that\n# also VERBATIM_HEADERS is set to NO.\n# The default value is: NO.\n\nSOURCE_BROWSER = NO\n\n# Setting the INLINE_SOURCES tag to YES will include the body of functions,\n# classes and enums directly into the documentation.\n# The default value is: NO.\n\nINLINE_SOURCES = NO\n\n# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any\n# special comment blocks from generated source code fragments. Normal C, C++ and\n# Fortran comments will always remain visible.\n# The default value is: YES.\n\nSTRIP_CODE_COMMENTS = YES\n\n# If the REFERENCED_BY_RELATION tag is set to YES then for each documented\n# function all documented functions referencing it will be listed.\n# The default value is: NO.\n\nREFERENCED_BY_RELATION = NO\n\n# If the REFERENCES_RELATION tag is set to YES then for each documented function\n# all documented entities called/used by that function will be listed.\n# The default value is: NO.\n\nREFERENCES_RELATION = NO\n\n# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set\n# to YES then the hyperlinks from functions in REFERENCES_RELATION and\n# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will\n# link to the documentation.\n# The default value is: YES.\n\nREFERENCES_LINK_SOURCE = YES\n\n# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the\n# source code will show a tooltip with additional information such as prototype,\n# brief description and links to the definition and documentation. Since this\n# will make the HTML file larger and loading of large files a bit slower, you\n# can opt to disable this feature.\n# The default value is: YES.\n# This tag requires that the tag SOURCE_BROWSER is set to YES.\n\nSOURCE_TOOLTIPS = YES\n\n# If the USE_HTAGS tag is set to YES then the references to source code will\n# point to the HTML generated by the htags(1) tool instead of doxygen built-in\n# source browser. The htags tool is part of GNU's global source tagging system\n# (see http://www.gnu.org/software/global/global.html). You will need version\n# 4.8.6 or higher.\n#\n# To use it do the following:\n# - Install the latest version of global\n# - Enable SOURCE_BROWSER and USE_HTAGS in the config file\n# - Make sure the INPUT points to the root of the source tree\n# - Run doxygen as normal\n#\n# Doxygen will invoke htags (and that will in turn invoke gtags), so these\n# tools must be available from the command line (i.e. in the search path).\n#\n# The result: instead of the source browser generated by doxygen, the links to\n# source code will now point to the output of htags.\n# The default value is: NO.\n# This tag requires that the tag SOURCE_BROWSER is set to YES.\n\nUSE_HTAGS = NO\n\n# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a\n# verbatim copy of the header file for each class for which an include is\n# specified. Set to NO to disable this.\n# See also: Section \\class.\n# The default value is: YES.\n\nVERBATIM_HEADERS = YES\n\n#---------------------------------------------------------------------------\n# Configuration options related to the alphabetical class index\n#---------------------------------------------------------------------------\n\n# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all\n# compounds will be generated. Enable this if the project contains a lot of\n# classes, structs, unions or interfaces.\n# The default value is: YES.\n\nALPHABETICAL_INDEX = YES\n\n# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in\n# which the alphabetical index list will be split.\n# Minimum value: 1, maximum value: 20, default value: 5.\n# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.\n\nCOLS_IN_ALPHA_INDEX = 3\n\n# In case all classes in a project start with a common prefix, all classes will\n# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag\n# can be used to specify a prefix (or a list of prefixes) that should be ignored\n# while generating the index headers.\n# This tag requires that the tag ALPHABETICAL_INDEX is set to YES.\n\nIGNORE_PREFIX =\n\n#---------------------------------------------------------------------------\n# Configuration options related to the HTML output\n#---------------------------------------------------------------------------\n\n# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output\n# The default value is: YES.\n\nGENERATE_HTML = YES\n\n# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a\n# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of\n# it.\n# The default directory is: html.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_OUTPUT = html\n\n# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each\n# generated HTML page (for example: .htm, .php, .asp).\n# The default value is: .html.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_FILE_EXTENSION = .html\n\n# The HTML_HEADER tag can be used to specify a user-defined HTML header file for\n# each generated HTML page. If the tag is left blank doxygen will generate a\n# standard header.\n#\n# To get valid HTML the header file that includes any scripts and style sheets\n# that doxygen needs, which is dependent on the configuration options used (e.g.\n# the setting GENERATE_TREEVIEW). It is highly recommended to start with a\n# default header using\n# doxygen -w html new_header.html new_footer.html new_stylesheet.css\n# YourConfigFile\n# and then modify the file new_header.html. See also section \"Doxygen usage\"\n# for information on how to generate the default header that doxygen normally\n# uses.\n# Note: The header is subject to change so you typically have to regenerate the\n# default header when upgrading to a newer version of doxygen. For a description\n# of the possible markers and block names see the documentation.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_HEADER = @abs_top_srcdir@/docs/head.html\n\n# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each\n# generated HTML page. If the tag is left blank doxygen will generate a standard\n# footer. See HTML_HEADER for more information on how to generate a default\n# footer and what special commands can be used inside the footer. See also\n# section \"Doxygen usage\" for information on how to generate the default footer\n# that doxygen normally uses.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_FOOTER = @abs_top_srcdir@/docs/foot.html\n\n# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style\n# sheet that is used by each HTML page. It can be used to fine-tune the look of\n# the HTML output. If left blank doxygen will generate a default style sheet.\n# See also section \"Doxygen usage\" for information on how to generate the style\n# sheet that doxygen normally uses.\n# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as\n# it is more robust and this tag (HTML_STYLESHEET) will in the future become\n# obsolete.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_STYLESHEET = @abs_top_srcdir@/docs/typical.css\n\n# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined\n# cascading style sheets that are included after the standard style sheets\n# created by doxygen. Using this option one can overrule certain style aspects.\n# This is preferred over using HTML_STYLESHEET since it does not replace the\n# standard style sheet and is therefore more robust against future updates.\n# Doxygen will copy the style sheet files to the output directory.\n# Note: The order of the extra style sheet files is of importance (e.g. the last\n# style sheet in the list overrules the setting of the previous ones in the\n# list). For an example see the documentation.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_EXTRA_STYLESHEET =\n\n# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or\n# other source files which should be copied to the HTML output directory. Note\n# that these files will be copied to the base HTML output directory. Use the\n# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these\n# files. In the HTML_STYLESHEET file, use the file name only. Also note that the\n# files will be copied as-is; there are no commands or markers available.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_EXTRA_FILES =\n\n# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen\n# will adjust the colors in the style sheet and background images according to\n# this color. Hue is specified as an angle on a colorwheel, see\n# http://en.wikipedia.org/wiki/Hue for more information. For instance the value\n# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300\n# purple, and 360 is red again.\n# Minimum value: 0, maximum value: 359, default value: 220.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_HUE = 60\n\n# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors\n# in the HTML output. For a value of 0 the output will use grayscales only. A\n# value of 255 will produce the most vivid colors.\n# Minimum value: 0, maximum value: 255, default value: 100.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_SAT = 13\n\n# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the\n# luminance component of the colors in the HTML output. Values below 100\n# gradually make the output lighter, whereas values above 100 make the output\n# darker. The value divided by 100 is the actual gamma applied, so 80 represents\n# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not\n# change the gamma.\n# Minimum value: 40, maximum value: 240, default value: 80.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_COLORSTYLE_GAMMA = 100\n\n# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML\n# page will contain the date and time when the page was generated. Setting this\n# to NO can help when comparing the output of multiple runs.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_TIMESTAMP = YES\n\n# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML\n# documentation will contain sections that can be hidden and shown after the\n# page has loaded.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_DYNAMIC_SECTIONS = NO\n\n# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries\n# shown in the various tree structured indices initially; the user can expand\n# and collapse entries dynamically later on. Doxygen will expand the tree to\n# such a level that at most the specified number of entries are visible (unless\n# a fully collapsed tree already exceeds this amount). So setting the number of\n# entries 1 will produce a full collapsed tree by default. 0 is a special value\n# representing an infinite number of entries and will result in a full expanded\n# tree by default.\n# Minimum value: 0, maximum value: 9999, default value: 100.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nHTML_INDEX_NUM_ENTRIES = 100\n\n# If the GENERATE_DOCSET tag is set to YES, additional index files will be\n# generated that can be used as input for Apple's Xcode 3 integrated development\n# environment (see: http://developer.apple.com/tools/xcode/), introduced with\n# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a\n# Makefile in the HTML output directory. Running make will produce the docset in\n# that directory and running make install will install the docset in\n# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at\n# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html\n# for more information.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_DOCSET = NO\n\n# This tag determines the name of the docset feed. A documentation feed provides\n# an umbrella under which multiple documentation sets from a single provider\n# (such as a company or product suite) can be grouped.\n# The default value is: Doxygen generated docs.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_FEEDNAME = \"Doxygen generated docs\"\n\n# This tag specifies a string that should uniquely identify the documentation\n# set bundle. This should be a reverse domain-name style string, e.g.\n# com.mycompany.MyDocSet. Doxygen will append .docset to the name.\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_BUNDLE_ID = org.doxygen.Project\n\n# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify\n# the documentation publisher. This should be a reverse domain-name style\n# string, e.g. com.mycompany.MyDocSet.documentation.\n# The default value is: org.doxygen.Publisher.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_PUBLISHER_ID = org.doxygen.Publisher\n\n# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher.\n# The default value is: Publisher.\n# This tag requires that the tag GENERATE_DOCSET is set to YES.\n\nDOCSET_PUBLISHER_NAME = Publisher\n\n# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three\n# additional HTML index files: index.hhp, index.hhc, and index.hhk. The\n# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop\n# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on\n# Windows.\n#\n# The HTML Help Workshop contains a compiler that can convert all HTML output\n# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML\n# files are now used as the Windows 98 help format, and will replace the old\n# Windows help format (.hlp) on all Windows platforms in the future. Compressed\n# HTML files also contain an index, a table of contents, and you can search for\n# words in the documentation. The HTML workshop also contains a viewer for\n# compressed HTML files.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_HTMLHELP = NO\n\n# The CHM_FILE tag can be used to specify the file name of the resulting .chm\n# file. You can add a path in front of the file if the result should not be\n# written to the html output directory.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nCHM_FILE =\n\n# The HHC_LOCATION tag can be used to specify the location (absolute path\n# including file name) of the HTML help compiler (hhc.exe). If non-empty,\n# doxygen will try to run the HTML help compiler on the generated index.hhp.\n# The file has to be specified with full path.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nHHC_LOCATION =\n\n# The GENERATE_CHI flag controls if a separate .chi index file is generated\n# (YES) or that it should be included in the master .chm file (NO).\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nGENERATE_CHI = NO\n\n# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc)\n# and project file content.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nCHM_INDEX_ENCODING =\n\n# The BINARY_TOC flag controls whether a binary table of contents is generated\n# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it\n# enables the Previous and Next buttons.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nBINARY_TOC = NO\n\n# The TOC_EXPAND flag can be set to YES to add extra items for group members to\n# the table of contents of the HTML help documentation and to the tree view.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTMLHELP is set to YES.\n\nTOC_EXPAND = NO\n\n# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and\n# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that\n# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help\n# (.qch) of the generated HTML documentation.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_QHP = NO\n\n# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify\n# the file name of the resulting .qch file. The path specified is relative to\n# the HTML output folder.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQCH_FILE =\n\n# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help\n# Project output. For more information please see Qt Help Project / Namespace\n# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace).\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_NAMESPACE = org.doxygen.Project\n\n# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt\n# Help Project output. For more information please see Qt Help Project / Virtual\n# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual-\n# folders).\n# The default value is: doc.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_VIRTUAL_FOLDER = doc\n\n# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom\n# filter to add. For more information please see Qt Help Project / Custom\n# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-\n# filters).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_CUST_FILTER_NAME =\n\n# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the\n# custom filter to add. For more information please see Qt Help Project / Custom\n# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-\n# filters).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_CUST_FILTER_ATTRS =\n\n# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this\n# project's filter section matches. Qt Help Project / Filter Attributes (see:\n# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes).\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHP_SECT_FILTER_ATTRS =\n\n# The QHG_LOCATION tag can be used to specify the location of Qt's\n# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the\n# generated .qhp file.\n# This tag requires that the tag GENERATE_QHP is set to YES.\n\nQHG_LOCATION =\n\n# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be\n# generated, together with the HTML files, they form an Eclipse help plugin. To\n# install this plugin and make it available under the help contents menu in\n# Eclipse, the contents of the directory containing the HTML and XML files needs\n# to be copied into the plugins directory of eclipse. The name of the directory\n# within the plugins directory should be the same as the ECLIPSE_DOC_ID value.\n# After copying Eclipse needs to be restarted before the help appears.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_ECLIPSEHELP = NO\n\n# A unique identifier for the Eclipse help plugin. When installing the plugin\n# the directory name containing the HTML and XML files should also have this\n# name. Each documentation set should have its own identifier.\n# The default value is: org.doxygen.Project.\n# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES.\n\nECLIPSE_DOC_ID = org.doxygen.Project\n\n# If you want full control over the layout of the generated HTML pages it might\n# be necessary to disable the index and replace it with your own. The\n# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top\n# of each HTML page. A value of NO enables the index and the value YES disables\n# it. Since the tabs in the index contain the same information as the navigation\n# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nDISABLE_INDEX = YES\n\n# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index\n# structure should be generated to display hierarchical information. If the tag\n# value is set to YES, a side panel will be generated containing a tree-like\n# index structure (just like the one that is generated for HTML Help). For this\n# to work a browser that supports JavaScript, DHTML, CSS and frames is required\n# (i.e. any modern browser). Windows users are probably better off using the\n# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can\n# further fine-tune the look of the index. As an example, the default style\n# sheet generated by doxygen has an example that shows how to put an image at\n# the root of the tree instead of the PROJECT_NAME. Since the tree basically has\n# the same information as the tab index, you could consider setting\n# DISABLE_INDEX to YES when enabling this option.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nGENERATE_TREEVIEW = YES\n\n# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that\n# doxygen will group on one line in the generated HTML documentation.\n#\n# Note that a value of 0 will completely suppress the enum values from appearing\n# in the overview section.\n# Minimum value: 0, maximum value: 20, default value: 4.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nENUM_VALUES_PER_LINE = 4\n\n# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used\n# to set the initial width (in pixels) of the frame in which the tree is shown.\n# Minimum value: 0, maximum value: 1500, default value: 250.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nTREEVIEW_WIDTH = 250\n\n# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to\n# external symbols imported via tag files in a separate window.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nEXT_LINKS_IN_WINDOW = NO\n\n# Use this tag to change the font size of LaTeX formulas included as images in\n# the HTML documentation. When you change the font size after a successful\n# doxygen run you need to manually remove any form_*.png images from the HTML\n# output directory to force them to be regenerated.\n# Minimum value: 8, maximum value: 50, default value: 10.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nFORMULA_FONTSIZE = 10\n\n# Use the FORMULA_TRANPARENT tag to determine whether or not the images\n# generated for formulas are transparent PNGs. Transparent PNGs are not\n# supported properly for IE 6.0, but are supported on all modern browsers.\n#\n# Note that when changing this option you need to delete any form_*.png files in\n# the HTML output directory before the changes have effect.\n# The default value is: YES.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\nFORMULA_TRANSPARENT = YES\n\n# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see\n# http://www.mathjax.org) which uses client side Javascript for the rendering\n# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX\n# installed or if you want to formulas look prettier in the HTML output. When\n# enabled you may also need to install MathJax separately and configure the path\n# to it using the MATHJAX_RELPATH option.\n# The default value is: NO.\n# This tag requires that the tag GENERATE_HTML is set to YES.\n\n# Slows page loading too much\nUSE_MATHJAX = NO\n\n# When MathJax is enabled you can set the default output format to be used for\n# the MathJax output. See the MathJax site (see:\n# http://docs.mathjax.org/en/latest/output.html) for more details.\n# Possible values are: HTML-CSS (which is slower, but has the best\n# compatibility), NativeMML (i.e. MathML) and SVG.\n# The default value is: HTML-CSS.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_FORMAT = HTML-CSS\n\n# When MathJax is enabled you need to specify the location relative to the HTML\n# output directory using the MATHJAX_RELPATH option. The destination directory\n# should contain the MathJax.js script. For instance, if the mathjax directory\n# is located at the same level as the HTML output directory, then\n# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax\n# Content Delivery Network so you can quickly see the result without installing\n# MathJax. However, it is strongly recommended to install a local copy of\n# MathJax from http://www.mathjax.org before deployment.\n# The default value is: http://cdn.mathjax.org/mathjax/latest.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest\n\n# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax\n# extension names that should be enabled during MathJax rendering. For example\n# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_EXTENSIONS =\n\n# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces\n# of code that will be used on startup of the MathJax code. See the MathJax site\n# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an\n# example see the documentation.\n# This tag requires that the tag USE_MATHJAX is set to YES.\n\nMATHJAX_CODEFILE =\n\n# When the SEARCHENGINE tag is enabled doxygen will generate a search box for\n# the HTML output. The underlying search engine uses javascript and DHTML and\n# should work on any modern browser. Note that when using HTML help\n# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET)\n# there is already a search function so this one should typically be disabled.\n# For large projects the javascript based search engine can be slow, then\n# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to\n# search using the keyboard; to jump to the search box use + S\n# (what the is depends on the OS and browser, but it is typically\n# , /