Repository: tbanel/orgaggregate
Branch: master
Commit: a6454ecdb2e3
Files: 15
Total size: 492.3 KB
Directory structure:
gitextract_cw3af3tx/
├── .gitignore
├── LICENSE
├── README.org
├── orgtbl-aggregate.el
├── orgtbl-aggregate.info
└── tests/
├── distant-tests.org
├── geography-a.csv
├── geography-a.json
├── hline-hash.json
├── hline-header.json
├── hline.csv
├── unfoldtest.org
├── unittests.org
├── wizard-test.el
└── wizardtests.org
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*~
*.elc
================================================
FILE: LICENSE
================================================
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc.
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{one line to give the program's name and a brief idea of what it does.}
Copyright (C) {year} {name of author}
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
{project} Copyright (C) {year} {fullname}
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
.
================================================
FILE: README.org
================================================
# -*- mode: org; coding:utf-8; -*-
#+TITLE: Aggregate Values in a Table
#+OPTIONS: ^:{} authors:Thierry Banel, Michael Brand toc:nil
Aggregating a table is creating a new table by computing sums,
averages, and so on, out of material from the first table.
* New
Transpose-babel blocks now handle =@#= and =hline= special columns. =@#= is
the input table row number. =hline= is the block number between two
horizontal lines where the current row is located.
And by the way, yes, =orgtbl-aggregate= comes with =orgtbl-transpose= as a
bonus. It flips rows with columns.
* Table of Contents
:PROPERTIES:
:TOC: :include siblings :depth 2 :force () :ignore (this) :local (nothing)
:CUSTOM_ID: table-of-contents
:END:
:CONTENTS:
- [[#examples][Examples]]
- [[#a-very-simple-example][A very simple example]]
- [[#demonstrate-sum-and-average-computing][Demonstrate sum and average computing]]
- [[#example-without-days][Example without days]]
- [[#example-of-counting-each-combination][Example of counting each combination]]
- [[#stop-reading-here-8020][Stop reading here! 80/20]]
- [[#name-your-input-table][Name your input table]]
- [[#create-an-aggregation-block][Create an aggregation block]]
- [[#refresh-the-aggregation][Refresh the aggregation]]
- [[#equivalent-in-sql-r-datamash-el-tblfn-awk-c][Equivalent in SQL, R, Datamash, el-tblfn, Awk, C++]]
- [[#sql-equivalent][SQL equivalent]]
- [[#r-equivalent][R equivalent]]
- [[#datamash-equivalent][Datamash equivalent]]
- [[#el-tblfn][el-tblfn]]
- [[#awk-equivalent][Awk equivalent]]
- [[#c-equivalent][C++ equivalent]]
- [[#wizards][Wizards]]
- [[#guiding-traditional-wizard][Guiding (traditional) wizard]]
- [[#experimental-free-form-wizard][Experimental free form wizard]]
- [[#the-cols-parameter][The :cols parameter]]
- [[#names-of-input-columns][Names of input columns]]
- [[#grouping-specifications-in-cols][Grouping specifications in :cols]]
- [[#the-hline-column][The hline column]]
- [[#the--column][The @# column]]
- [[#aggregation-formulas-in-cols][Aggregation formulas in :cols]]
- [[#correlation-of-two-columns][Correlation of two columns]]
- [[#almost-any-expression-can-be-specified][(Almost) any expression can be specified]]
- [[#column-names][Column names]]
- [[#input-table-with-or-without-a-header][Input table with or without a header]]
- [[#column-names-of-the-input-table][Column names of the input table]]
- [[#multiple-lines-header][Multiple lines header]]
- [[#custom-column-names][Custom column names]]
- [[#formatters][Formatters]]
- [[#org-mode-compatible-formatters][Org Mode compatible formatters]]
- [[#debugging-formatters][Debugging formatters]]
- [[#discarding-an-output-column][Discarding an output column]]
- [[#sorting][Sorting]]
- [[#example-with-one-sorting-column][Example with one sorting column]]
- [[#several-sorting-columns][Several sorting columns]]
- [[#hlines-in-the-output-table][hlines in the output table]]
- [[#output-hlines-depends-on-sorting-columns][Output hlines depends on sorting columns]]
- [[#example-with-hline-2][Example with hline 2]]
- [[#cells-processing][Cells processing]]
- [[#where-calc-interpretation-happens][Where Calc interpretation happens?]]
- [[#dates][Dates]]
- [[#durations][Durations]]
- [[#empty-and-malformed-input-cells][Empty and malformed input cells]]
- [[#symbolic-computation][Symbolic computation]]
- [[#intervals][Intervals]]
- [[#error-or-precision-forms][Error or precision forms]]
- [[#wide-variety-of-inputs][Wide variety of inputs]]
- [[#standard-org-mode-input][Standard Org Mode input]]
- [[#virtual-input-table-from-babel][Virtual input table from Babel]]
- [[#an-org-id][An Org ID]]
- [[#csv-input][CSV input]]
- [[#json-input][JSON input]]
- [[#input-slicing][Input slicing]]
- [[#the-cond-filter][The :cond filter]]
- [[#virtual-input-columns][Virtual input columns]]
- [[#post-processing][Post-processing]]
- [[#spreadsheet-formulas][Spreadsheet formulas]]
- [[#algorithm-post-processing][Algorithm post processing]]
- [[#grand-total][Grand total]]
- [[#chaining][Chaining]]
- [[#pull--push][Pull & Push]]
- [[#pull-mode][Pull mode]]
- [[#push-mode][Push mode]]
- [[#pull-or-push-][Pull or push ?]]
- [[#debugging][Debugging]]
- [[#seeing-the--forms][Seeing the $ forms]]
- [[#seeing-calc-formulas-before-evaluation][Seeing Calc formulas before evaluation]]
- [[#seeing-lisp-internal-form-of-calc-formulas][Seeing Lisp internal form of Calc formulas]]
- [[#example-of-debugging-vsumnn2][Example of debugging vsum(nn^2)]]
- [[#summary-of-debugging-formatters][Summary of debugging formatters]]
- [[#tricks][Tricks]]
- [[#sorting-0][Sorting]]
- [[#a-few-lowest-or-highest-values][A few lowest or highest values]]
- [[#span-of-values][Span of values]]
- [[#no-aggregation][No aggregation]]
- [[#installation][Installation]]
- [[#authors-contributors][Authors, contributors]]
- [[#changes][Changes]]
- [[#gpl-3-license][GPL 3 License]]
:END:
* Examples
:PROPERTIES:
:CUSTOM_ID: examples
:END:
** A very simple example
:PROPERTIES:
:CUSTOM_ID: a-very-simple-example
:END:
We have a table of activities and quantities (whatever they are) over
several days.
#+begin_example
#+name: original
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
#+end_example
To begin with we want to gather all colors and count how many times
they appear. We are interested only in the second column named =Color=
First we give a name to the table through the =#+NAME:=
or =#+TBLNAME:= tags, just above the table.
Then we create a /dynamic block/ to receive the aggregation:
#+begin_example
desired output columns╶──────────────────────────╮
the input table╶──────────────╮ │
type of processing╶─╮ │ │
╭───────╯ │ │
▼ ╭───┴────╮ ╭─────┴───────╮
#+begin: aggregate :table "original" :cols "Color count()"
#+end:
#+end_example
Now typing =C-c C-c= in the dynamic block counts the colors in the original table:
#+begin_example
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "Color count()"
| Color | count() |
|-------+---------|
| Red | 7 |
| Blue | 7 |
#+end:
#+end_example
OrgAggregate found two colors, =Red= and =Blue=. It found 7 occurrences
for each.
** Demonstrate sum and average computing
:PROPERTIES:
:CUSTOM_ID: demonstrate-sum-and-average-computing
:END:
Now we want to aggregate this table for each day (because several rows
exist for each day). We want the average value of the =Level= column for
each day, and the sum of the =Quantity= column. We write down the
block specifying that (later we will see how to automate the creation
of such a block with a [[#wizards][wizard]]):
#+begin_example
sum aggregation╶─────────────────────────────────────────────────╮
average aggregation╶───────────────────────────────╮ │
key grouping column╶───────────────────────╮ │ │
╭┴╮ ╭────┴─────╮ ╭─────┴──────╮
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
#+end
#+end_example
Typing =C-c C-c= in the dynamic block computes the aggregation:
#+begin_example
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
╰┬╯ ╰────┬─────╯ ╰──────┬─────╯
╭───────────────────────────────────────╯ │ │
│ ╭───────────────────────────────╯ │
│ │ ╭──────────────────────────────╯
╭┴╮ ╭────┴─────╮ ╭─────┴──────╮
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+end
#+end_example
The source table is not changed in any way.
To get this result, we specified columns in this way, after the
=:cols= parameter:
- =Day= : we got the same column as in the source table, except
entries are not duplicated. Here =Day= acts as a /key grouping column/.
We may specify as many key columns as we want just by naming them.
We get only one aggregated row for each different combination
of values of key grouping columns.
- =vmean(Level)= : this instructs OrgAggregate to compute the average of
values found in the =Level= column, grouped by the same =Day=.
- =vsum(Quantity)=: OrgAggregate computes the sum of values found in the
=Quantity= column, one sum for each =Day=.
** Example without days
:PROPERTIES:
:CUSTOM_ID: example-without-days
:END:
Maybe we are just interested in the sum of =Quantities=, regardless of
=Days=. We just type:
#+begin_example
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "vsum(Quantity)"
╰─────┬──────╯
╭──────────────────────────────────────────╯
╭────┴───────╮
| vsum(Quantity) |
|----------------|
| 218 |
#+end
#+end_example
** Example of counting each combination
:PROPERTIES:
:CUSTOM_ID: example-of-counting-each-combination
:END:
we may want to count the number of rows for each combination of
=Day= and =Color=:
#+BEGIN_EXAMPLE
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+BEGIN: aggregate :table "original" :cols "count() Day Color"
╰──┬──╯ ╰┬╯ ╰─┬─╯
╭─────────────────────────────────────────╯ │ │
│ ╭───────────────────────────────────────╯ │
│ │ ╭───────────────────────────────╯
╭──┴──╮ ╭┴╮ ╭─┴─╮
| count() | Day | Color |
|---------+-----------+-------|
| 1 | Monday | Red |
| 1 | Monday | Blue |
| 2 | Tuesday | Red |
| 1 | Tuesday | Blue |
| 1 | Wednesday | Red |
| 2 | Wednesday | Blue |
| 3 | Thursday | Red |
| 3 | Friday | Blue |
#+END
#+END_EXAMPLE
If we want to get measurements for =Colors= rather than =Days=, we
type:
#+begin_example
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "Color vmean(Level) vsum(Quantity)"
╰─┬─╯ ╰────┬─────╯ ╰─────┬──────╯
╭─────────────────────────────────────────╯ │ │
│ ╭──────────────────────────────────────╯ │
│ │ ╭────────────────────────────────────╯
╭─┴─╮ ╭────┴─────╮ ╭─────┴──────╮
| Color | vmean(Level) | vsum(Quantity) |
|-------+---------------+----------------|
| Red | 40.2857142857 | 144 |
| Blue | 15.5714285714 | 74 |
#+end
#+end_example
* Stop reading here! 80/20
:PROPERTIES:
:CUSTOM_ID: stop-reading-here-8020
:END:
If you managed to get here, you are at 80/20 (thanks Pareto!). You
grasped only 20% of the OrgAggregate features, but those 20% cover 80%
of the use cases.
To summarize the 20%:
** Name your input table
:PROPERTIES:
:CUSTOM_ID: name-your-input-table
:END:
- Select one of your Org table, and be ready to aggregate values
from it right in the same file.
- Give a name to your table with a special line just above it.
#+begin_example
name here╶───╮
╭────╯
▼
#+name: original
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
…
#+end_example
** Create an aggregation block
:PROPERTIES:
:CUSTOM_ID: create-an-aggregation-block
:END:
#+begin_example
#+begin: aggregate
#+end:
#+end_example
- *Input:* specify a =:table= parameter.
- *Output:* specify the desired output columns with the =:cols= parameter.
#+begin_example
output╶───────────────────────────────────────────╮
input╶───────────────────────╮ │
│ │
╭──┴───╮ ╭────────┴─────────╮
#+begin: aggregate :table original :cols "Day vsum(Quantity)"
#+end:
#+end_example
** Refresh the aggregation
:PROPERTIES:
:CUSTOM_ID: refresh-the-aggregation
:END:
- Type =C-c C-c= on the =#+begin:= line now and whenever you want to
refresh the aggregation.
#+begin_example
C-c C-c here╶─╮
│
▼
#+begin: aggregate :table original :cols "Day vsum(Quantity)"
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Tuesday | 45 |
…
#+end:
#+end_example
* Equivalent in SQL, R, Datamash, el-tblfn, Awk, C++
:PROPERTIES:
:CUSTOM_ID: equivalent-in-sql-r-datamash-el-tblfn-awk-c
:END:
Aggregation is a widely used method to get insights in tabular
data. Use whatever environment best suits your needs.
OrgAggregate is great when you want to output and Org Mode
table. Also, OrgAggregate has no dependency other than Emacs, not even
other Lisp packages.
** SQL equivalent
:PROPERTIES:
:CUSTOM_ID: sql-equivalent
:END:
If you are familiar with SQL, you would get a similar result with the
=GROUP BY= statement:
#+begin_src sql
select Day, mean(Level), sum(Quantity)
from original
group by Day;
#+end_src
** R equivalent
:PROPERTIES:
:CUSTOM_ID: r-equivalent
:END:
If you are familiar with the R statistical language, you would get a
similar result with =factor= and =aggregate= functions:
#+begin_src R
original <- the table as a data.frame
day_factor <- factor(original$Day)
aggregate (original$Level , list(Day=day_factor), mean)
aggregate (original$Quantity, list(Day=day_factor), sum )
#+end_src
** Datamash equivalent
:PROPERTIES:
:CUSTOM_ID: datamash-equivalent
:END:
The command-line Datamash software operates on CSV files and can
achieve a similar result:
#+begin_src shell
datamash -H -g Day mean Level sum Quantity 1 {
Day =$1
Color =$2
Level =$3
Quantity =$4
SumLevel[Day] += Level
SumQuantity[Day] += Quantity
Count[Day] ++
}
END {
for (d in SumQuantity) {
printf "%s %s %s\n", d, SumLevel[d]/Count[d], SumQuantity[d]
}
}
#+end_src
#+end_example
** C++ equivalent
:PROPERTIES:
:CUSTOM_ID: c-equivalent
:END:
C++ has hash-maps in its standard template library. And Org Mode
provides support for C++ Babel blocks. Thus, it is quite
straigthforward to aggegrate in this language.
(Don't forget to customize =org-src-lang-modes= to activate C++ support
in Org Mode).
#+begin_example
#+begin_src C++ :var original=original :includes '( )
using namespace std;
unordered_map SumLevel;
unordered_map SumQuantity;
unordered_map Count;
for (auto row : original) {
auto Day = row[0];
auto Color = row[1];
auto Level = stod(row[2]);
auto Quantity = stod(row[3]);
SumLevel[Day] += Level;
SumQuantity[Day] += Quantity;
Count[Day] ++;
}
for (auto it : SumQuantity)
cout<= so that it does not appear in the output.
#+begin_example
invisible╶─────────────────────────────────────────╮
sorted numerically decreasing╶───────────────────╮ │
row numbers of input table╶──────────────────╮ │ │
│ │ │
▼ ▼ ▼
#+begin: aggregate :table "original" :cols "Day Color Level Quantity @#;^N;<>"
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Friday | Blue | 11 | 9 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 7 | 5 |
| Thursday | Red | 49 | 30 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 39 | 24 |
| Wednesday | Blue | 15 | 15 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Red | 27 | 23 |
| Tuesday | Blue | 33 | 18 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Red | 51 | 12 |
| Monday | Blue | 25 | 3 |
| Monday | Red | 30 | 11 |
#+end:
#+end_example
** Aggregation formulas in :cols
:PROPERTIES:
:CUSTOM_ID: aggregation-formulas-in-cols
:END:
Aggregation formulas are applied for each of the groupings, on the
specified columns.
We saw examples with =sum=, =mean=, =count= aggregations. There are
many other aggregations. They are based on functions provided by Calc
(Calc is the powerful Emacs calculator):
- =count()= or =vcount()=
+ in Calc: =`u #' (`calc-vector-count') [`vcount'])=
+ gives the number of elements in the group being aggregated;
this function may or may not take a column parameter;
with a parameter, empty cells are not counted
(except with the =E= modifier)..
- =sum(X)= or =vsum(X)=
+ in Calc: =`u +' (`calc-vector-sum') [`vsum']=
+ computes the sum of elements being aggregated
- =cnorm(X)=
+ in Calc: =`v N' (calc-cnorm') [`cnorm']=
+ like =vsum(X)=, compute the sum of values, but first replacing negative
values by their opposite
- =max(X)= or =vmax(X)=
+ in Calc: =`u X' (`calc-vector-max') [`vmax']=
+ gives the largest of the elements being aggregated
- =min(X)= or =vmin(X)=
+ in Calc: =`u N' (`calc-vector-min') [`vmin']=
+ gives the smallest of the elements being aggregated
- =span(X)= or =vspan(X)=
+ in Calc: =`v :' (`calc-set-span') [`vspan']=
+ summarizes values to be aggregated into an interval =[MIN..MAX]=
where =MIN= and =MAX= are the minimal and maximal values to be aggregated
- =rnorm(X)=
+ in Calc: =`v n' (`calc-rnorm) [`rnorm']=
+ like =vmax(X)=, gives the maximum of values, but first replacing negative
values by their opposite
- =mean(X)= or =vmean(X)=
+ in Calc: =`u M' (`calc-vector-mean') [`vmean']=
+ computes the average (arithmetic mean) of elements being aggregated
- =meane(X)= or =vmeane(X)=
+ in Calc: =`I u M' (`calc-vector-mean-error') [`vmeane']=
+ computes the average (as mean) along with the estimated error of elements being aggregated
- =median(X)= or =vmedian(X)=
+ in Calc: =`H u M' (`calc-vector-median') [`vmedian']=
+ computes the median of elements being aggregated, by taking the middle element after sorting them
- =hmean(X)= or =vhmean(X)=
+ in Calc: =`H I u M' (`calc-vector-harmonic-mean') [`vhmean']=
+ computes the harmonic mean of elements being aggregated
- =gmean(X)= or =vgmean(X)=
+ in Calc: =`u G' (`calc-vector-geometric-mean') [`vgmean']=
+ computes the geometric mean of elements being aggregated
- =sdev(X)= or =vsdev(X)=
+ in Calc: =`u S' (`calc-vector-sdev') [`vsdev']=
+ computes the standard deviation of elements being aggregated
- =psdev(X)= or =vpsdev(X)=
+ in Calc: =`I u S' (`calc-vector-pop-sdev') [`vpsdev']=
+ computes the population standard deviation (divide by N instead of N-1)
- =var(X)= or =vvar(X)=
+ in Calc: =`H u S' (`calc-vector-variance') [`vvar']=
+ computes the variance of elements being aggregated
- =pvar(X)= or =vpvar(X)=
+ in Calc: =`H u S' (`calc-vector-variance') [`vpvar']=
+ computes the population variance of elements being aggregated
- =pcov(X,Y)= or =vpcov(X,Y)=
+ in Calc: =`I u C' (`calc-vector-pop-covariance') [`vpcov']=
+ computes the population covariance of elements being aggregated from two columns (divides by N)
- =cov(X,Y)= or =vcov(X,Y)=
+ in Calc: =`u C' (`calc-vector-covariance') [`vcov']=
+ computes the sample covariance of elements being aggregated from two columns (divides by N-1)
- =corr(X,Y)= or =vcorr(X,Y)=
+ in Calc: =`H u C' (`calc-vector-correlation') [`vcorr']=
+ computes the linear correlation coefficient of elements being aggregated in two columns
- =prod(X)= or =vprod(X)=
+ in Calc: =`u *' (`calc-vector-product') [`vprod']=
+ computes the product of elements being aggregated
- =vlist(X)= or =list(X)=
+ gives the list of =X= being aggregated, verbatim, without aggregation.
- =(X)= or =X= in a formula
+ returns the list of =X= being aggregated, without aggregation,
passed through Calc interpretation.
- =sort(X)=
+ in Calc: =`v S' (`calc-sort') [`sort']=
+ sorts elements to be aggregated in ascending order;
only works on numerical values
- =rsort(X)=
+ in Calc: =`I v S' (`calc-sort') [`sort']=
+ sorts elements to be aggregated in descending order;
only works on numerical values
- =rev(X)=
+ in Calc: =`' (`calc-reverse-vector') [`rev']=
+ returns the list of values to be aggregated in reverse order
- =subvec(X,from)=, =subvec(X,from,to)=
+ in Calc: =`v s' (`calcFunc-subvec') [`subvec']=
+ extracts a sub-list from =X= starting at =from= and ending at =to= excluded
(or up to the end if =to= is not given).
The first value is numbered =1=. So for instance
=subvec(X,1,3)= extracts the first two values
- =vmask(M,X)=
+ in Calc: =`v m' (`calcFunc-vmask') [`vmask']=
+ extracts a sub-list from =X=, keeping only values for which corresponding values in
=M= (the mask) are not zero
- =head(X)=
+ in Calc: =`v h' (`calc-head') [`head']=
+ returns the first value to be aggregated
- =rtail(X)=
+ in Calc: =`H I v h' (`calc-head') [`rtail']=
+ returns the last value to be aggregated
- =find(X,val)=
+ in Calc: =`v f' (`calc-vector-find') [`find']=
+ returns the index of =val= in the list of values to be aggregated, or =0=
if =val= is not found. Index starts from =1=
- =rdup(X)=
+ in Calc: =`v +' (`calc-remove-duplicates') [`rdup']=
+ remove duplicates from =X= and returns remaining values sorted in
ascending order
- =grade(X)=
+ in Calc: =`v G' (`calc-grade') [`grade']=
+ returns a list of index of values to be aggregated: the index of the lowest value,
then the second lowest value, and so on up to the index of the highest value.
Indexes start from =1=
- =rgrade(X)=
+ in Calc: =`I v G' (`calc-grade') [`rgrade']=
+ Like =grade= in reverse order
The aggregation functions may be written with or without a leading
=v=. =sum= and =vsum= are equivalent. The =v= form should be
preferred, as it is the one used in the Org table spreadsheet, and in
Calc. The non-v names may be dropped in the future.
** Correlation of two columns
:PROPERTIES:
:CUSTOM_ID: correlation-of-two-columns
:END:
Some aggregations work on two columns (rather than one column for
=vsum()=, =vmean()=).
Those aggregations are =vcov(,)=, =vpcov(,)=, =vcorr(,)=.
- =vcorr(,)= computes the linear correlation between two columns.
- =vcov(,)= and =vpcov(,)= compute the covariance of two columns.
Example. We create a table where column =y= is a noisy version of
column =x=:
#+begin_example
#+tblname: noisydata
| bin | x | y |
|-------+----+---------|
| small | 1 | 10.454 |
| small | 2 | 21.856 |
| small | 3 | 30.678 |
| small | 4 | 41.392 |
| small | 5 | 51.554 |
| large | 6 | 61.824 |
| large | 7 | 71.538 |
| large | 8 | 80.476 |
| large | 9 | 90.066 |
| large | 10 | 101.070 |
| large | 11 | 111.748 |
| large | 12 | 121.084 |
#+tblfm: $3=$2*10+random(1000)/500;%.3f
#+end_example
#+begin_example
#+BEGIN: aggregate :table noisydata :cols "bin vcorr(x,y) vcov(x,y) vpcov(x,y)"
| bin | vcorr(x,y) | vcov(x,y) | vpcov(x,y) |
|-------+----------------+---------------+---------------|
| small | 0.999459736649 | 25.434 | 20.3472 |
| large | 0.999542438688 | 46.4656666667 | 39.8277142857 |
#+END
#+end_example
We see that the correlation between =x= and =y= is very close to =1=,
meaning that both columns are correlated. Indeed they are, as the =y=
is computed from =x= with the formula
#+begin_example
y = 10*x + noise_between_0_and_2
#+end_example
** (Almost) any expression can be specified
:PROPERTIES:
:CUSTOM_ID: almost-any-expression-can-be-specified
:END:
Virtually any Calc formula can be specified as an aggregation formula.
Single column name (as they appear in the header of the source table,
or in the form of =$1=, =$2=, ..., or the virtual columns =hline= and
=@#=) are key columns. Everything else is given to Calc, to be
computed as an aggregation.
For instance:
#+begin_example
(3) ;; a constant
vmean(2*X+1) ;; aggregate an expression
exp(vmean(map(log,N))) ;; the exponential average
vsum((X-vmean(X))^2) ;; X-vmean(X) centers the sample on zero
#+end_example
Arguably, the first expression is useless, but legal. The aggregation
can be applied to a computed list of values. The result of an
aggregation can be further processed in a formula. An aggregation can
even be applied to an expression containing another aggregation.
In an expression, if a variable has the name of a column, then it is
replaced by a Calc vector containing values from this column.
The special expression =(C)= (a column name within parenthesis)
yields a list of values to be aggregated from this column, except they
are not aggregated. Note that parenthesis are required, otherwise, =C=
would act as a key grouping column.
* Column names
:PROPERTIES:
:CUSTOM_ID: column-names
:END:
** Input table with or without a header
:PROPERTIES:
:CUSTOM_ID: input-table-with-or-without-a-header
:END:
The header of a table gives names to its columns. It is separated from
data with an horizontal line.
#+begin_example
column name is "quantity" or "$2"╶╮
column name is "day" or "$1"╶╮ │
╭─────────────────────────╯ │
│ ╭───────────────╯
▼ ▼
| day | quantity |
|-----------+----------|
| monday | 12.3 |
| monday | 5.9 |
| thursday | 41.1 |
| wednesday | 16.8 |
#+end_example
In this example, the input columns may be referred to as =day= and
=quantity=.
Tables without a header are handled by OrgAggregate with /"dollar
names"/. Example of a table without a header:
#+begin_example
column name is "$2"╶──╮
column name is "$1"╶╮ │
╭───────────────╯ │
│ ╭─╯
▼ ▼
| monday | 12.3 |
| monday | 5.9 |
| thursday | 41.1 |
| wednesday | 16.8 |
#+end_example
Then columns may be refereed to as =$1= and =$2=.
** Column names of the input table
:PROPERTIES:
:CUSTOM_ID: column-names-of-the-input-table
:END:
Column names are not necessarily alphanumeric words. They may contain
any characters, including spaces, quotes, +, -, whatever. They must
not extend on several lines thought.
Those names need to be protected with quotes (single or double quotes)
within formulas.
Examples:
- =:cols= "=mean('estimated value')="
- =:cond (equal "true color" "Red")=
Quoting is not required for
- ASCII letters
- numbers
- underscore _, dollar $, dot .
- accented letters like à é
- Greek letters like α, Ω
- northern letters like ø
- Russian letters like й
- Esperanto letters like ŭ
- Japanese ideograms like 量
Note that in =:cond= Lisp expression, only double quotes work. This is
because single quotes in Lisp have a very special meaning.
=Ubuntu Mono= font can be used for displaying aligned Japanese
characters, although not perfectly.
** Multiple lines header
:PROPERTIES:
:CUSTOM_ID: multiple-lines-header
:END:
The header of the source table may be more than one row tall. Only the
first header row is used to match column names between the source
table and the =:cols= specifications.
Best effort is made to propagate additional header rows to the
aggregated table. This happens when the aggregated column refers to a
single source column, either as a key column or a formula involving a
single column.
#+begin_example
#+name: tall-header
╭──────────────────────────────────────────────╮
╭───┴──╮ │
| color | quantity | level | ╶╮ │
| | | <3> | ├─╴header is 3 rows tall │
| kolor | kiom | nivelo | ╶╯ │
|--------+----------+--------| ╭───────────╯───────────────╮
| yellow | 72 | 3 | │ only the first header row │
| green | 55 | 5 | │ is used in formulas │
| | | | ╰───────────╭───────────────╯
| orange | 80 | 2 | │
| yellow | 13 | 1 | │
╭───┴──╮
#+BEGIN: aggregate :table "tall-header" :cols "color vsum(quantity);'sum' count();'nb' vsum(quantity)/vmean(level);'leveled'"
| color | sum | nb | leveled | ╶╮
| | | | | ├──╮
| kolor | kiom | | | ╶╯ │ ╭────────────────────────╮
|--------+------+----+---------| │ │ best attempt to recover│
| yellow | 85 | 2 | 42.5 | ╰──┤ the three header rows │
| green | 55 | 1 | 11 | │ in the output │
| orange | 80 | 1 | 40 | ╰────────────────────────╯
#+END:
#+end_example
Note that the last aggregated column has just =leveled= in its header.
This is because this column refers to more than one source columns,
namely =quantity= and =level=.
Note that in this example, there are formatting cookies:
: <> <7>
Data rows containing at least one cookie are ignored. They are not
ignored in the header.
** Custom column names
:PROPERTIES:
:CUSTOM_ID: custom-column-names
:END:
In this example, output column have names which are difficult to
handle:
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vmean(Level*2) vsum(Quantity^2)"
╰─────┬──────╯ ╰──────┬───────╯
╭───────────────────────────────╯ │
│ ╭─────────────────────────────╯
╭─────┴──────╮ ╭──────┴───────╮
| Day | vmean(Level*2) | vsum(Quantity^2) |
|-----------+----------------+------------------|
| Monday | 55 | 130 |
| Tuesday | 86 | 693 |
| Wednesday | 36 | 1010 |
| Thursday | 86 | 2317 |
| Friday | 16 | 170 |
#+END
#+end_example
We can give them custom names with the =;'custom name'= decoration:
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vmean(Level*2);'mean2' vsum(Quantity^2);'sum_squares'"
╰──┬──╯ ╰─────┬─────╯
╭───────────────────────────────────────────────╯ │
│ ╭─────────────────────────────────────────────────────────────────╯
╭─┴─╮ ╭───┴─────╮
| Day | mean2 | sum_squares |
|-----------+-------+-------------|
| Monday | 55 | 130 |
| Tuesday | 86 | 693 |
| Wednesday | 36 | 1010 |
| Thursday | 86 | 2317 |
| Friday | 16 | 170 |
#+END
#+end_example
Decorators are optional.
* Formatters
:PROPERTIES:
:CUSTOM_ID: formatters
:END:
** Org Mode compatible formatters
:PROPERTIES:
:CUSTOM_ID: org-mode-compatible-formatters
:END:
An expression may optionally be followed by modifiers and formatters,
after a semicolon. Examples:
#+begin_example
vsum(X);p20 ;; increase Calc internal precision to 20 digits
vsum(X);f3 ;; output the result with 3 digits after the decimal dot
vsum(X);%.3f ;; output the result with 3 digits after the decimal dot
#+end_example
The modifiers and formatters are fully compatible with those of the
Org Mode spreadsheet.
- =p12= change the precision to 12 decimal digits.
- =n7= output as floating point number with 7 decimal digits.
- =f4= output number with 4 decimal places after dot.
- =s5= output number in "scientific" mode (with exponent of 10) with 5
decimal digits.
- =e6= output number in "engineering" mode (with exponent of 10 multiple
of 3) with 6 decimal digits.
- =t= output duration in decimal hours; input is supposed to be either a
duration like ="2:37"= meaning 2 hours and 37 minutes, or a number of
seconds like ="1234=" which is approximately =0.34= hours. The output is
controlled by the =org-table-duration-custom-format= variable.
- =T= output duration in an hours-minutes-seconds format like ="01:20:34"=
meaning 1 hour, 20 minutes, and 34 seconds.
- =U= like =t=, but disregard the =org-table-duration-custom-format=
variable and use =hh:mm= in place.
- =N= output number: remove any non-numeric output.
- =E= keep empty input cells. The result is often =nan=. Without =E=, empty
input cells are ignored as if they did not exist.
- =D= angles are in degrees.
- =R= angles are in radians.
- =F= output is presented as a fraction of integers if it actually
is. The format is the Calc one, for example ="2:3"= means =2/3=.
- =S= symbolic mode. When an input cell is, for instance =sqrt(2)=, it it
kept as-is rather than being replaced by =1.41421=.
** Debugging formatters
:PROPERTIES:
:CUSTOM_ID: debugging-formatters
:END:
Additionally, a few formatters are dedicated to debugging:
- =c= output the Calc expression before substitution by actual input
cells values.
- =q= output the Lisp expression before substitution by actual input
cells values.
- =C= output the Calc expression before it gets simplified and folded.
- =Q= output the Lisp expression before it gets simplified and folded.
See [[#debugging][Debugging]] for a detailed explanation.
** Discarding an output column
:PROPERTIES:
:CUSTOM_ID: discarding-an-output-column
:END:
Why would anyone specify a column just to discard it in the output? For
its side effects. For sorting the output table or for adding hlines to
it.
To discard a column, add a =;<>= modifier to the column
description. This syntax is reminiscent of the == cookies in Org Mode
tables, which instructs to shorten a column width to only =n=
characters.
In this example, input hlines create a =hline= column which is used to
add hlines to the output. Then this =hline= column is discarded with =<>=.
#+begin_example
invisible╶────────────────────────────────────────────╮
sorted numerically increasing╶──────────────────────╮ │
│ │
▼ ▼
#+BEGIN: aggregate :table "withhline" :cols "hline;^n;<> cölØr vsum(vâluε)" :hline 1
| cölØr | vsum(vâluε) |
|--------+-------------|
| Red | 7.4 |
| Yellow | 9.1 |
|--------+-------------|
| Blue | 15.7 |
| Yellow | 5.4 |
|--------+-------------|
| Blue | 4.9 |
| Red | 3.9 |
| Yellow | 9. |
|--------+-------------|
| Red | 1.1 |
| Yellow | 3.4 |
#+END:
#+end_example
Here is an example where rows are sorted on the =cölØr= column, but
without displaying this column:
#+begin_example
invisible╶────────────────────────────────────────────╮
sorted alphabetically╶──────────────────────────────╮ │
│ │
▼ ▼
#+BEGIN: aggregate :table "withhline" :cols "cölØr;^a;<> vâluε;^n" :hline 1
| vâluε | ▲
|-------| │
| 4.9 | within the same cölØr bucket, │
| 7.0 | sort vâluε numerically╶─────────────────────╯
| 8.7 |
|-------|
| 1.1 |
| 1.3 |
| 2.6 |
| 3.5 |
| 3.9 |
|-------|
| 2.4 |
| 3.4 |
| 5.4 |
| 6.6 |
| 9.1 |
#+END:
#+end_example
* Sorting
:PROPERTIES:
:CUSTOM_ID: sorting
:END:
** Example with one sorting column
:PROPERTIES:
:CUSTOM_ID: example-with-one-sorting-column
:END:
In this example, the output table is sorted numerically on its second
column (look at the =^n= specification):
#+begin_example
#+begin: aggregate :table "original" :cols "Day vsum(Quantity);^n"
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Friday | 22 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
#+end:
#+end_example
By default, no sorting is done. The output rows follows the ordering
of the input rows.
Any column specification in the =:cols= parameter may be followed by a
semicolon and caret characters, and an ordering.
The specification for the ordering are the same as in Org Mode:
- =a=: ascending alphabetical sort
- =A=: descending alphabetical sort
- =n=: ascending numerical sort
- =N=: descending numerical sort
- =t=: ascending date, time, or duration sort
- =T=: descending date, time, or duration sort
- =f= & =F= specifications are not (yet) implemented
** Several sorting columns
:PROPERTIES:
:CUSTOM_ID: several-sorting-columns
:END:
The rows of the resulting table may be sorted on any combination of
its columns.
Several columns may get a sorting specification. The major column is
used for sorting. Only when two rows are equal regarding the major
column, the second major column is compared. And if the two rows are
still equal on this second column, the third is used, and so on.
The first sorted column in the =:cols= parameter is the major one. To
declare another one as the major, follow it with a number, for
instance =1=. Columns without a number are minor ones.
Example:
#+begin_example
:cols "AAA;^a BBB;^N2 CCC DDD;^t1"
╭────╮ ▲╷ ▲▲ ▲▲ ╭──────────────╮
│sort╰──────╯╰─╮ ││ │╰─╯first priority│
│alphabetically│ ││ ╰──╮sort by date │
│third priority│ ││ ╰──────────────╯
╰──────────────╯ ││ ╭────────────────╮
│╰──╯second priority │
╰───╮sort numerically│
│decreasing │
╰────────────────╯
#+end_example
- Column =DDD= is sorted in ascending dates or times (=t=
specification). It is the major sorting column (because of its =1=
numbering).
- Column =BBB= sorts rows which compare equal on column =DDD= (because of
its =2= numbering). This column is assumed to contain numerical
values, and it is sorted in descending order (=N= specification).
- Column =AAA= is used to sort rows which compare equal regarding =DDD=
and =BBB=. It is sorted in ascending alphabetical order (=a=
specification).
Both a format and a sorting instruction may be given. Example:
#+begin_example
:cols "EXPR:f3:^n"
#+end_example
The =EXPR= column is
- formatted with 3 digits after dot (=f3=)
- sorted numerically in ascending order (=^n=).
* hlines in the output table
:PROPERTIES:
:CUSTOM_ID: hlines-in-the-output-table
:END:
The =:hline N= parameter controls horizontal lines in the output
table. It may or may not be related to horizontal lines in the input.
** Output hlines depends on sorting columns
:PROPERTIES:
:CUSTOM_ID: output-hlines-depends-on-sorting-columns
:END:
Example of an input table:
#+begin_example
#+name: withouthline
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
#+end_example
Horizontal lines appear on the sorted column, which in this example is
the =cölØr= column.
We require output hlines with =:hline 1=. The =1= value here says that
only one sorted column should be considered when drawing output
horizontal lines. A value of =2= would mean to consider two sorted
columns.
Horizontal lines will separate blocks of identical =cölØr= rows:
#+begin_example
#+BEGIN: aggregate :table "withouthline" :cols "cölØr;^a vâluε 'ra;han'" :hline 1
| cölØr | vâluε | 'ra;han' | ╰─┬─╯ ▲ ╰─┬─╯
|--------+-------+----------| │ │ │
| Blue | 8.7 | 52 |╶╮ │ │╭──────╮ ╭─────────┴─────╮
| Blue | 7.0 | 29 | ├─╴Blue bucket │ ╰╯sort │ │ separate │
| Blue | 4.9 | 64 |╶╯ ╰─────╮cölØr │ │ cölØr buckets │
|--------+-------+----------| ◀────────────────╮ ╰──────╯ │ with hlines │
| Red | 1.3 | 41 |╶╮ │ ╰──┬┬───────────╯
| Red | 3.5 | 35 | │ ╰─────────────────────╯│
| Red | 2.6 | 84 | ├─╴Red bucket │
| Red | 3.9 | 51 | │ │
| Red | 1.1 | 58 |╶╯ │
|--------+-------+----------| ◀───────────────────────────────────────╯
| Yellow | 9.1 | 95 |╶╮
| Yellow | 5.4 | 17 | │
| Yellow | 2.4 | 55 | ├─╴Yellow bucket
| Yellow | 6.6 | 34 | │
| Yellow | 3.4 | 51 |╶╯
#+END:
#+end_example
** Example with hline 2
:PROPERTIES:
:CUSTOM_ID: example-with-hline-2
:END:
In the following example, we specify =:hline 2=.
First, the input table now have horizontal lines. We want to propagate
them to the output.
#+begin_example
#+name: withhline
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
|--------+-------+--------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
|--------+-------+--------|
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+--------|
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
#+end_example
The two sorted columns are =hline= and =cölØr=. Therefore output
horizontal lines separate blocks of identical =hline= and =cölØr=:
#+begin_example
#+begin: aggregate :table "withhline" :cols "hline;^n cölØr;^a vâluε 'ra;han'" :hline 2
| hline | cölØr | vâluε | 'ra;han' | ▲ ▲ ▲
|-------+--------+-------+----------| │ │ │
| 0 | Red | 1.3 | 41 | ╭──────╯ ╰───────────╮ │
| 0 | Red | 3.5 | 35 | │ two sorted output columns │ │
| 0 | Red | 2.6 | 84 | ╰───────────────────────────╯ │
|-------+--------+-------+----------| │
| 0 | Yellow | 9.1 | 95 | │
|-------+--------+-------+----------| ╭─────────────────────────────╮ │
| 1 | Blue | 8.7 | 52 | │ 2 means: create hlines ├─────╯
| 1 | Blue | 7.0 | 29 | │ for buckets and sub-buckets │
|-------+--------+-------+----------| ╰─────────────────────────────╯
| 1 | Yellow | 5.4 | 17 |
|-------+--------+-------+----------|╶─╮
| 2 | Blue | 4.9 | 64 | │ ╭──────────────────────────╮
|-------+--------+-------+----------| ╶┤ │ bucket hline=2 │
| 2 | Red | 3.9 | 51 | ├───┤ divided in 3 sub-buckets │
|-------+--------+-------+----------| ╶┤ │ Blue, Red, Yellow │
| 2 | Yellow | 2.4 | 55 | │ ╰──────────────────────────╯
| 2 | Yellow | 6.6 | 34 | │
|-------+--------+-------+----------|╶─╯
| 3 | Red | 1.1 | 58 |
|-------+--------+-------+----------|
| 3 | Yellow | 3.4 | 51 |
#+end:
#+end_example
And the =hline= column may be discarded (but its side effect
remains). To do so use the =;<>= specifier:
#+begin_example
#+begin: aggregate :table "withhline" :cols "hline;^n;<> cölØr;^a vâluε 'ra;han'" :hline 2
| cölØr | vâluε | 'ra;han' |
|--------+-------+----------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Red | 2.6 | 84 |
|--------+-------+----------|
| Yellow | 9.1 | 95 |
|--------+-------+----------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
|--------+-------+----------|
| Yellow | 5.4 | 17 |
|--------+-------+----------|
| Blue | 4.9 | 64 |
|--------+-------+----------|
| Red | 3.9 | 51 |
|--------+-------+----------|
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+----------|
| Red | 1.1 | 58 |
|--------+-------+----------|
| Yellow | 3.4 | 51 |
#+end:
#+end_example
The =:hline= parameter accepts a number:
- =:hline 0=, =:hline no=, =:hline nil=, or no =:hline= mean that there will
be no hlines in the output.
- =:hline 1=, =:hline yes=, =:hline t= mean that hlines will separate blocks
of identical rows regarding the major sorted column. In case no
column is sorted, then output hlines will reflect input ones.
- =:hline 2= means that the major and the next major sorted columns will
be used to separate identical rows regarding those two columns.
- =:hline 3=, =:hline 4=, ... may be specified, but they may result in too
much hlines.
* Cells processing
:PROPERTIES:
:CUSTOM_ID: cells-processing
:END:
*Calc* is the standard Emacs desktop calculator. Actual mathematical
computations are handled through Calc. This offers a lot of
flexibility.
** Where Calc interpretation happens?
:PROPERTIES:
:CUSTOM_ID: where-calc-interpretation-happens
:END:
Example of input table. Besides numbers, there are cells with
mathematical expressions like =20*30=, or just labels as =Red&Green=
without any mathematical meaning.
#+begin_example
#+name: to_Calc_or_not_to_Calc
| Day | Color | Level |
|-----------+------------+--------|
| Monday | Red | 20*30 |
| Monday | Blue | 55+45 |
| Tuesday | Red | 1 |
| Tuesday | Red&Green | 2 |
| Tuesday | Blue+Green | 3 |
| Wednesday | Red | (27) |
| Wednesday | Red | (12+1) |
| Wednesday | Green | [15] |
#+end_example
Basically, Calc operates twice. For example in the formula
=vsum(Level)=:
- Calc computes =Level= for every input cell in the =Level= column,
- then Calc computes =vsum()= applied to the resulting list.
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vsum(Level)"
| Day | vsum(Level) |
|-----------+-------------|
| Monday | 700 |
| Tuesday | 6 |
| Wednesday | 55 |
#+END:
#+end_example
There are a few occasions were Calc computation does not happen:
=vcount()= and =vlist(X)=.
The =vcount()= sub-formula is evaluated as the number of input rows in
each group, without Calc intervention. However, later on Calc can
handle this number in a formula as this one: =vsum(Level)/vcount()=
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vcount() vsum(Level)/vcount()"
| Day | vcount() | vsum(Level)/vcount() |
|-----------+----------+----------------------|
| Monday | 2 | 350 |
| Tuesday | 3 | 2 |
| Wednesday | 3 | 18.333333 |
#+END:
#+end_example
And of course when input cells do not have a mathematical meaning, the
result may be non-sens:
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vsum(Color)"
| Day | vsum(Color) |
|-----------+------------------------------------------------|
| Monday | Red + Blue |
| Tuesday | Red + error(3, '"Syntax error") + Blue + Green |
| Wednesday | 2 Red + Green |
#+END:
#+end_example
But it can also make sens. The last row, which aggregate =Wednesday=
rows, is computed as =2 Red + Green=. This is correct. This symbolic
result (as opposed to numerical result) shows the power of Calc as a
symbolic calculator.
The =vlist(X)= formula is not handled by Calc at all. This formula
must appear alone (not embedded as part of a bigger formula). The cells
=X= are not interpreted by Calc. As a result, =vlist(X)= produces a
cell which concatenates input cells verbatim. For instance, the input
cell =20*30= is left as-is.
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vlist(Color) vlist(Level)"
| Day | vlist(Color) | vlist(Level) |
|-----------+----------------------------+--------------------|
| Monday | Red, Blue | 20*30, 55+45 |
| Tuesday | Red, Red&Green, Blue+Green | 1, 2, 3 |
| Wednesday | Red, Red, Green | (27), (12+1), [15] |
#+END:
#+end_example
As a contrast, the formula =(Level)= yields a list processed through
Calc. For instance, the =20*30= formula is replaced by =600=.
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day (Color) (Level)"
| Day | (Color) | (Level) |
|-----------+------------------------------------------------+----------------|
| Monday | [Red, Blue] | [600, 100] |
| Tuesday | [Red, error(3, '"Syntax error"), Blue + Green] | [1, 2, 3] |
| Wednesday | [Red, Red, Green] | [27, 13, [15]] |
#+END:
#+end_example
Here we used parenthesis in =(Color)= and =(Level)= because otherwise
they would have been /key columns/. Instead of parenthesis, we can
embed such expressions in formulas, like =Level+1=:
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day Level+1"
| Day | Level+1 |
|-----------+----------------|
| Monday | [601, 101] |
| Tuesday | [2, 3, 4] |
| Wednesday | [28, 14, [16]] |
#+END:
#+end_example
To summarize, a column name embedded in a formula is evaluated as the
list of input cells, processed by Calc. Except for the =vlist(Column)=
formula where input cells are kept verbatim.
By the way, what is the meaning of the expression =Level*Level=? For
=Monday=, it is =[600,100]*[600,100]=. Then Calc simplifies that as a
/vector product/: sum of individual products. =600^2+100^2=
#+begin_example
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day Level*Level Level+Level"
| Day | Level*Level | Level+Level |
|-----------+-------------+----------------|
| Monday | 370000 | [1200, 200] |
| Tuesday | 14 | [2, 4, 6] |
| Wednesday | 1123 | [54, 26, [30]] |
#+END:
#+end_example
** Dates
:PROPERTIES:
:CUSTOM_ID: dates
:END:
Some aggregations are possible on dates. Example. Here is a source
table containing dates:
#+begin_example
#+tblname: datetable
| Date |
|------------------------|
| [2035-12-22 Sat 09:01] |
| [2034-11-24 Fri 13:04] |
| [2030-09-24 Tue 13:54] |
| [2027-09-25 Sat 03:54] |
| [2023-02-26 Sun 16:11] |
| [2020-03-17 Tue 03:51] |
| [2018-08-21 Tue 00:00] |
| [2012-12-25 Tue 00:00] |
#+end_example
Here are the earliest and the latest dates, along with the average of
all input dates:
#+begin_example
#+BEGIN: aggregate :table datetable :cols "vmin(Date) vmax(Date) vmean(Date)"
| vmin(Date) | vmax(Date) | vmean(Date) |
|------------------------+------------------------+-------------|
| <2012-12-25 Tue 00:00> | <2035-12-22 Sat 09:01> | 739448.44 |
#+END:
#+end_example
The average of all dates is a number? Actually, it is a date expressed
as the number of days since =[0000-12-31 Sun 00:00]=. To force a
number of days to be interpreted as a date, use the =date()= function:
#+begin_example
#+BEGIN: aggregate :table datetable :cols "date(vmean(Date))"
| date(vmean(Date)) |
|------------------------|
| <2025-07-16 Wed 10:29> |
#+END:
#+end_example
With the =date()= function in mind, all kinds of dates handling can be
done. Example: the average of earliest and the latest dates is
different from the average of all dates:
#+begin_example
#+BEGIN: aggregate :table datetable :cols "date(vmean(vmin(Date),vmax(Date))) date(vmean(Date))"
| date(vmean(vmin(Date),vmax(Date))) | date(vmean(Date)) |
|------------------------------------+------------------------|
| <2024-06-23 Sun 16:30> | <2025-07-16 Wed 10:29> |
#+END:
#+end_example
Note that =date()= is not special to OrgAggregate. It can be used in
Org Mode spreadsheet formulas.
** Durations
:PROPERTIES:
:CUSTOM_ID: durations
:END:
In Org Mode spreadsheet, durations have the forms =HH:MM= or
=HH:MM:SS=. In OrgAggregate, when an input cell have one of those two
forms, it is converted into a number of seconds. For instance, =01:00=
is converted into =3600= and =00:00:07= is converted into =7=.
There may be a single digit for hours, as in =7:12= or more than two
as in =1255:45:00=.
To output such a form, use a formatter: =;T=; =;t=, =;U=. For example, we
have 3 durations as input, and we want the average of them:
#+begin_example
#+name: some_durations
| dur |
|----------|
| 07:45:30 |
| 13:55 |
| 17:12 |
#+end_example
#+begin_example
#+BEGIN: aggregate :table "some_durations" :cols "vmean(dur) vmean(dur);T vmean(dur);t vmean(dur);U"
| vmean(dur) | vmean(dur) | vmean(dur) | vmean(dur) |
|------------+------------+------------+------------|
| 46650 | 12:57:30 | 12.96 | 12:57 |
#+END:
#+end_example
- With no formatter, we get a number of seconds
- The =T= formatter outputs the result as =HH:MM:SS=
- The =U= formatter outputs the result as =HH:MM=
- The =t= formatter converts the result into a number of hours (it
divides the number of seconds by 3600, and displays only two digits
after dot)
The Calc syntax for durations is also recognized:
#+begin_example
HH@ MM'
HH@ MM' SS"
#+end_example
Example:
#+begin_example
#+name: calc_durations
| dur |
|------------|
| 07@ 45' 30 |
| 13@ 55' |
| 17@ 12' |
#+end_example
#+begin_example
#+BEGIN: aggregate :table "calc_durations" :cols "vmean(dur)"
| vmean(dur) |
|--------------|
| 12@ 57' 30." |
#+END:
#+end_example
** Empty and malformed input cells
:PROPERTIES:
:CUSTOM_ID: empty-and-malformed-input-cells
:END:
The input table may contain malformed mathematical text. For
instance, a cell containing =5+= is malformed, because an expression
is missing after the =+= symbol. In this case, the value will be
replaced by =error(2, '"Expected a number")= which will appear in the
aggregated table, signaling the problem.
An input cell may be empty. In this case, it may be ignored or
converted to zero, depending on modifier flags =E= and =N=.
The empty cells treatment
- makes no difference for =vsum= and =count=.
- may result in zero for =prod=,
- change =vmean= result,
- change =vmin= and =vmax=, a possibly empty list of values resulting in
=inf= or =-inf=
Some aggregation functions operate on two columns. If the two columns
have empty values at different locations, then they should be
interpreted as zero with the =NE= modifier, otherwise the result will
be inconsistent.
Sometimes an input table may be malformed, with incomplete rows, like
this one:
#+begin_example
| Color | Level | Quantity | Day |
|-------+-------+----------+-----------|
| Red | 30 | 11 | Monday |
| Blue | 25 | 3 | Monday |
|
| Blue | 33 | 18 | Tuesday |
| Red | 27 |
| Blue | 12 | 16 | Wednesday |
| Blue | 15 | 15 |
|
#+end_example
Missing cells are handled as though they were empty.
** Symbolic computation
:PROPERTIES:
:CUSTOM_ID: symbolic-computation
:END:
The computations are based on Calc, which is a symbolic calculator.
Thus, symbolic computations are built-in. Example:
This is the source table:
#+begin_example
#+NAME: symtable
| Day | Color | Level | Quantity |
|-----------+-------+--------+----------|
| Monday | Red | 30+x | 11+a |
| Monday | Blue | 25+3*x | 3 |
| Tuesday | Red | 51+2*x | 12 |
| Tuesday | Red | 45-x | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12+x | 16 |
| Wednesday | Blue | 15 | 15-6*a |
| Thursday | Red | 39 | 24-5*a |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49+x | 30+9*a |
| Friday | Blue | 7 | 5+a |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
#+end_example
And here is the aggregated, symbolic result:
#+begin_example
#+BEGIN: aggregate :table "symtable" :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+-----------------------+----------------|
| Monday | 2. x + 27.5 | a + 14 |
| Tuesday | 0.333333333334 x + 43 | 45 |
| Wednesday | x / 3 + 18 | 54 - 6 a |
| Thursday | x / 3 + 43. | 4 a + 83 |
| Friday | 8 | a + 22 |
#+END
#+end_example
Symbolic calculations are correctly performed on =x= and =a=, which
are symbolic (as opposed to numeric) expressions.
Note that if there are empty cells in the input, they will be changed
to =nan= /not a number/, and the whole aggregation will yield =nan=. This
is probably not the expected result.
The =N= modifier (see [[#org-mode-compatible-formatters][Org Mode compatible formatters]]) won't help,
because even though it will replace empty cells with zero, it will do
the same for anything which does not look like a number. The best is
to just avoid empty cells when dealing with symbolic calculations.
** Intervals
:PROPERTIES:
:CUSTOM_ID: intervals
:END:
Org Mode tables and OrgAggregate backend engine being Emacs Calc,
intervals are seamlessly handled. An interval is made of two numerical
values separated by two dots.
Example of a spreadsheet. The third column is computed by multiplying
the first two:
#+begin_example
,#+name: int
| 3..5 | 10 | (30 .. 50) |
| 3..5 | -1..2 | (-5 .. 10) |
| 5 | 2 | 10 |
,#+TBLFM: $3=$1*$2
#+end_example
Example of an aggregation. The sum of the third column is computed,
resulting in an interval:
#+begin_example
,#+BEGIN: aggregate :table "int" :cols "vsum($3)"
| vsum($3) |
|------------|
| (35 .. 70) |
,#+END:
#+end_example
** Error or precision forms
:PROPERTIES:
:CUSTOM_ID: error-or-precision-forms
:END:
In Emacs Calc, an error form is a numerical value along with its
precision, both values separated by =+/-=.
The separation may also be the Unicode character =±=. In this
spreadsheet example, the second column is 10 times the first:
#+begin_example
,#+name: cert
| 12±2 | 120 +/- 20 |
| 55±3 | 550 +/- 30 |
| 9±0 | 90 |
| 15±1 | 150 +/- 10 |
| 21±1 | 210 +/- 10 |
,#+TBLFM: $2=10*$1
#+end_example
Now, in the following example, OrgAggregate computes the sum of the
second column:
#+begin_example
,#+BEGIN: aggregate :table "cert" :cols "vsum($2);f2"
| vsum($2) |
|----------------|
| 1120 +/- 38.73 |
,#+END:
#+end_example
The computation made by Emacs Calc assumes that all input values
follow a Gaussian distribution, and are independent variables. Then it
applies the textbook formula for combining Gaussian distributions.
Beware that your input values may not be independent with each
other. In this case, the resulting error will be slightly off (too
small).
* Wide variety of inputs
:PROPERTIES:
:CUSTOM_ID: wide-variety-of-inputs
:END:
As in any other Org Mode source block, the input table may come from
several places. OrgAggregate adds even more kinds of input.
Here we discus the =:table= parameter.
** Standard Org Mode input
:PROPERTIES:
:CUSTOM_ID: standard-org-mode-input
:END:
The parameter after =:table= may be:
- =mytable=: an ordinary Org Mode table in the same buffer, named
=mytable=.
- =/some/dir/file.org:mytable=: an ordinary Org Mode table named
=mytable=, in a distant Org file named =/some/dir/file.org=.
** Virtual input table from Babel
:PROPERTIES:
:CUSTOM_ID: virtual-input-table-from-babel
:END:
- =mybabel=: an Org Mode Babel block named =mybabel= in the current
buffer, generating a table as its output, written in any language.
- =mybabel(param1=123,param2=456)=: passing parameters to an Org Mode
Babel block named =mybabel= in the current buffer, generating a table
as its output, written in any language.
- =/some/dir/file.org:mybabel(param1=123,param2=456)=: an Org Mode Babel
block named =mybabel= in a distant org file named =/some/dir/file.org=,
called with parameters.
The input table may be the result of executing a Babel script. In this
case, the table is virtual in the sense that is appears nowhere.
(Babel is the Org Mode infrastructure to run scripts in any language,
like Python, R, C++, Java, D, Rust, shell, whatever, with inputs and
outputs connected to Org Mode).
Example:
Here is a script in Emacs Lisp which creates an Org Mode table.
#+begin_example
#+name: ascript
#+begin_src elisp :colnames yes
`(
("label" value) ; cells are symbols or strings
hline
,@(cl-loop
for i from 10 to 20
collect
(list
(format "lbl-%c" (+ ?A (% i 3))) ; cell is a string
i))) ; cell is a number
#+end_src
#+end_example
If executed, the script would output this table:
#+begin_example
#+RESULTS: ascript
| label | value |
|-------+-------|
| lbl-B | 10 |
| lbl-C | 11 |
| lbl-A | 12 |
| lbl-B | 13 |
| lbl-C | 14 |
| lbl-A | 15 |
| lbl-B | 16 |
| lbl-C | 17 |
| lbl-A | 18 |
| lbl-B | 19 |
| lbl-C | 20 |
#+end_example
But instead, OrgAggregate will execute the script and consume its
output:
#+begin_example
#+BEGIN: aggregate :table "ascript" :cols "label vsum(value)"
| label | vsum(value) |
|-------+-------------|
| lbl-B | 58 |
| lbl-C | 62 |
| lbl-A | 45 |
#+END:
#+end_example
Here the parameter =:table= specifies the name of the script to be
executed.
*Beware* of the =:results code= parameter. It does not work as an input
for OrgAggregate. This is because in this case the Babel script
returns a string, not a table. Example:
#+begin_src elisp :results code
'((a b c)
hline
(1 2 3))
#+end_src
Use =:results table= or nothing instead.
** An Org ID
:PROPERTIES:
:CUSTOM_ID: an-org-id
:END:
- =34cbc63a-c664-471e-a620-d654b26ffa31=: an identifier of an Org Mode
sub-tree. The sub-tree is supposed to contain an Org table (which is
retrieved) or a Babel script (which is executed).
Those Org Mode identifiers span all known Org Mode files. (Therefore
there is no need to specify a file). To add such an identifier, put
the cursor on the heading of the sub-tree, and type =M-x
org-id-get-create=.
** CSV input
:PROPERTIES:
:CUSTOM_ID: csv-input
:END:
CVS input is specific to OrgAggregate. (Native Org Mode does not
offers those formats).
- =/some/dir/file.csv:(csv params…)=: a comma-separated-values file in
the CSV format, in the file =/some/dir/file.csv=.
- =name(csv params…)=: CSV formatted data within an Org block named
="name"=, in the current file.
- =/some/dir/file.org:name(csv params…)=: CSV formatted data within an
Org block named ="name"=, in a distant Org file.
The cells separators in the CSV files may be TAB, comma, or semicolon,
they are guessed and different separators may be mixed.
Any empty row in the CSV file is interpreted as an horizontal
separator (=hline= in Org table parlance).
Parameters may be:
- =header=: the first row in the CSV file is interpreted as a header
containing the column names.
- =colnames (column1 column2 column3 …)=: the column names are given
explicitly, in case the CSV file contains only data, no header.
In any case, the columns may be references as =$1, $2, $3, …= as
usual.
When data is in an Org Mode file, it is supposed to be within a named
block of any kind. Example with a "quote" block:
#+begin_example
#+name: mycsvdata
#+begin_quote
label,quantity
yellow,27
red,-61
yellow,41
red,-29
red,-17
#+end_quote
#+end_example
** JSON input
:PROPERTIES:
:CUSTOM_ID: json-input
:END:
- =/some/dir/file.json:(json params…)=: a file containing a JSON
formatted table, in the file =/some/dir/file.csv=.
- =name(json params…)=: JSON formatted data within an Org block named
="name"=, in the current file.
- =/some/dir/file.org:name(json params…)=: JSON formatted data within an
Org block named ="name"=, in a distant Org file.
The accepted formats are a vector of vectors, or a vector of
hash-objects.
Currently, no =params= are recognized.
Example of a vector of vectors:
#+begin_example
[
["mon",12,"red" ],
["thu",34,"blue" ],
["wed",27,"green"],
["wed",21,"red" ],
["mon", 7,"blue" ],
…
]
#+end_example
Example of a vector of hash-objects:
#+begin_example
[
{"day":"mon", "quty":12, "color":"red" },
{"day":"thu", "quty":34, "color":"blue" },
{"quty":27, "day":"wed", "color":"green"},
{"quty":21, "color":"red", "day":"wed" },
{"day":"mon", "quty": 7, "color":"blue" },
…
]
#+end_example
In the case of hash-objects, the keys are collected as the header of
the resulting table, in the order seen in the JSON file. In each
hash-object, the order of key-value pairs is irrelevant.
A header may be specified in the JSON file as a first vector, followed
by an hline (horizontal line). Example:
#+begin_example
[
["day","quty","color"],
"hline",
["mon",12,"red" ],
["thu",34,"blue" ],
…
]
#+end_example
Horizontal lines may be ="hline"=, =[]=, ={}=, or =null=.
It is possible to mix both formats: vectors and hash-objects. Example:
#+begin_example
[
["quty","color"], // incomplete header
null, // horizontal line
{"day":"mon", "quty":12, "color":"red" }, // an hash-object
["thu", 34, "blue" ], // a vector
…
]
#+end_example
When data is in an Org Mode file, it is supposed to be within a named
block of any kind. Example with a "src" block for JavaScript:
#+begin_example
#+name: myjsondata
#+begin_src js
[
["day","quty","color"],
"hline",
["mon",12,"red" ],
["thu",34,"blue" ],
…
]
#+end_src
#+end_example
** Input slicing
:PROPERTIES:
:CUSTOM_ID: input-slicing
:END:
Org Mode also provides for table slicing. All of the previous
references may be followed by an optional slicing. Examples:
- =mytable[0:5]=: retain only the first 6 rows of the input table; if
the table has a header, then it counts as 2 rows (the header and the
separation line). In this example, it would retain rows 0 and 1 for
the header, and rows 2,3,4,5 for the content.
- =mytable[*,0:1]=: retain only the first 2 columns.
- =mytable[0:5,0:1]=: retain only the first 6 rows and the first 2
columns.
** The :cond filter
:PROPERTIES:
:CUSTOM_ID: the-cond-filter
:END:
This parameter is optional. If present, it specifies a Lisp expression
which tells whether or not a row should be kept. When the expression
evaluates to nil, the row is discarded.
Examples of useful expressions includes:
- =:cond (equal Color "Red")=
+ to keep only rows where =Color= is =Red=
- =:cond (> (string-to-number Quantity) 19)=
+ to keep only rows for which =Quantity= is more than =19=
+ note the call to =string-to-number=; without this call, =Quantity=
would be used as a string
- =:cond (> (* (string-to-number Level) 2.5) (string-to-number Quantity))=
+ to keep only rows for which =2.5*Level > Quantity=
Beware with this example: =:cond (equal Color "Red")=. The input table
should not have a column named =Red=, otherwise the condition will mean:
/keep only rows with the same value in columns Color and Red/
As a special case, when =:cols= parameter is not given, the result is
the same as =:cols "COL1 COL2 COL3..."=. All columns in the input
table are specified as key columns, and output in the resulting table.
This is useful when just filtering. But be aware that aggregation
still occurs. So duplicate input rows appear only once in the result.
** Virtual input columns
:PROPERTIES:
:CUSTOM_ID: virtual-input-columns
:END:
What if we want to aggregate on months? But the input table contains
only plain dates. Example:
#+begin_example
#+name: without-months
| Date | Quantity |
|------------------+----------|
| [2037-03-12 thu] | 56.93 |
| [2037-03-25 wed] | 20.99 |
| [2037-04-07 tue] | 80.81 |
| [2037-04-20 mon] | 22.26 |
| [2037-05-03 sun] | 69.75 |
| [2037-05-16 sat] | 39.91 |
| [2037-05-29 fri] | 93.13 |
| [2037-06-11 thu] | 17.11 |
| [2037-06-24 wed] | 24.21 |
| [2037-07-07 tue] | 82.38 |
| [2037-07-20 mon] | 39.94 |
| [2037-08-02 sun] | 81.90 |
| [2037-08-15 sat] | 71.67 |
| [2037-08-28 fri] | 82.81 |
| [2037-09-10 thu] | 42.50 |
| [2037-09-23 wed] | 62.52 |
| [2037-10-06 tue] | 5.13 |
#+end_example
We would like to specify the aggregation over =month(Date)=. But only
plain columns may be used as aggregating keys.
One way is to add a =Month= column to the input table. The modified
table looks like:
#+begin_example
#+name: with-months
| Date | Quantity | Month |
|------------------+----------+-------|
| [2037-03-12 thu] | 56.93 | 3 |
| [2037-03-25 wed] | 20.99 | 3 |
| [2037-04-07 tue] | 80.81 | 4 |
| [2037-04-20 mon] | 22.26 | 4 |
| [2037-05-03 sun] | 69.75 | 5 |
| [2037-05-16 sat] | 39.91 | 5 |
| [2037-05-29 fri] | 93.13 | 5 |
| [2037-06-11 thu] | 17.11 | 6 |
| [2037-06-24 wed] | 24.21 | 6 |
| [2037-07-07 tue] | 82.38 | 7 |
| [2037-07-20 mon] | 39.94 | 7 |
| [2037-08-02 sun] | 81.90 | 8 |
| [2037-08-15 sat] | 71.67 | 8 |
| [2037-08-28 fri] | 82.81 | 8 |
| [2037-09-10 thu] | 42.50 | 9 |
| [2037-09-23 wed] | 62.52 | 9 |
| [2037-10-06 tue] | 5.13 | 10 |
#+TBLFM: $3=month($1)
#+end_example
OrgAggregate allows adding input columns like this computed =Month=
column, without modifying the input table. The =:precompute= parameter
does that. Example:
#+begin_example
#+BEGIN: aggregate :table "without-months" :cols "Month vsum(Quantity)" :precompute ("month(Date);'Month'")
| Month | vsum(Quantity) |
|-------+----------------|
| 3 | 77.92 |
| 4 | 103.07 |
| 5 | 202.79 |
| 6 | 41.32 |
| 7 | 122.32 |
| 8 | 236.38 |
| 9 | 105.02 |
| 10 | 5.13 |
#+END:
#+end_example
The specification =month(Date);'Month'= means:
- add a third column to the input table,
- fill it with the formula =month(Date)=, which is a Calc formula,
- give this new column the =Month= title,
- make this new column available for aggregation, as any other column.
All this process is virtual. The input table is not modified in any
way.
If the title =Month= is not specified, then the new virtual column will
be referred to as =$3=.
Note that here the title was specified with single quotes. This is
required to disambiguate with the format. The syntax is consistent
with the one used in the =:cols= parameter and the one used by Org Mode
spreadsheet formulas.
The pre-computations may also be Lisp expressions, exactly like in the
usual Org table spreadsheet. In this example, we want to aggregate on
coarse bins. Bins are just hundredths of the first column:
#+begin_example
#+name: want-bins
| 109 | 41.24 |
| 140 | 40.60 |
| 174 | 7.68 |
| 288 | 33.96 |
| 334 | 21.42 |
| 418 | 74.73 |
| 455 | 79.62 |
| 479 | 22.23 |
| 554 | 28.03 |
| 678 | 64.12 |
| 797 | 70.91 |
| 947 | 93.48 |
#+end_example
#+begin_example
#+BEGIN: aggregate :table "want-bins" :cols "$3 vmean($2)" :precompute ("'(floor (/ (string-to-number $1) 100))")
| $3 | vmean($2) |
|----+-----------|
| 1 | 29.84 |
| 2 | 33.96 |
| 3 | 21.42 |
| 4 | 58.86 |
| 5 | 28.03 |
| 6 | 64.12 |
| 7 | 70.91 |
| 9 | 93.48 |
#+END:
#+end_example
Virtual columns may be formatted as any other column, with the same
syntax as in =:cols= or in the Org table spreadsheet. For example here
we give it 2 digits after dot:
#+begin_example
#+BEGIN: aggregate :table "want-bins" :cols "$3" :precompute "floor($1/100);%.2f"
| $3 |
|------|
| 1.00 |
| 2.00 |
| 3.00 |
| 4.00 |
| 5.00 |
| 6.00 |
| 7.00 |
| 9.00 |
#+END:
#+end_example
Of course, those additional virtual input columns may be used for
other purposes than key columns. They may enter in aggregating
formulas. Or they may be used by the optional row filter (the =:cond=
parameter). There is no difference between actual and virtual columns.
The =:precompute= parameter may be:
- a list of strings, example:
: ("month(Date);'Month'" "day(Date);'Day'")
- a single string with fields separated by =::=, like in the =#+tblfm:=
tags of a spreadsheet. Example:
: "month(Date);'Month' :: day(Date);'Day'"
- a string containing a single formula (actually this is a special
case of the previous one). Example:
: "month(Date);'Month'"
* Post-processing
:PROPERTIES:
:CUSTOM_ID: post-processing
:END:
After OrgAggregate has generated the output table, it can be further
processed:
- additional columns may be added with the standard Org Mode
spreadsheet formulas.
- any algorithm in an exotic language (Python, R, C++, Emacs Lisp, and
so on) can be applied to the output.
** Spreadsheet formulas
:PROPERTIES:
:CUSTOM_ID: spreadsheet-formulas
:END:
Additional columns can be specified for the resulting table. With a
previous example, adding a =:formula= parameter, we specify a new column
=$4= which uses the aggregated columns. It is translated into a usual
=#+TBLFM:= spreadsheet line.
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)" :formula "$4=$2*$3"
| Day | vmean(Level) | vsum(Quantity) | |
|-----------+--------------+----------------+------|
| Monday | 27.5 | 14 | 385. |
| Tuesday | 43 | 45 | 1935 |
| Wednesday | 18 | 54 | 972 |
| Thursday | 43 | 83 | 3569 |
| Friday | 8 | 22 | 176 |
#+TBLFM: $4=$2*$3
#+END:
#+end_example
Moreover, if a =#+TBLFM:= was already there, it survives aggregation re-computations.
This happens in /pull mode/ only.
** Algorithm post processing
:PROPERTIES:
:CUSTOM_ID: algorithm-post-processing
:END:
The aggregated table can be post-processed with the =:post=
parameter. It accepts a Lisp =lambda=, a Lisp function, or a Babel block
in any exotic language (R, Python, C++, Emacs Lisp and so on).
The process receives the aggregated table as parameter in the form of
a Lisp expression. It can process it in any way it wants, provided it
returns a valid Lisp table.
A Lisp table is a list of rows. Each row is either a list of cells, or
the special symbol =hline=.
In this example, a =lambda= expression adds a =hline= and a row for /Sunday/.
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)" :post (lambda (table) (append table '(hline (Sunday "0.0"))))
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
|-----------+----------------|
| Sunday | 0.0 |
#+END:
#+end_example
The =lambda= can be moved to a =defun=. The function is then passed to the
=:post= parameter:
#+begin_example
,#+begin_src elisp
(defun my-function (table)
(append table
'(hline (Sunday "0.0"))))
,#+end_src
... :post my-function
#+end_example
The =:post= parameter can also refer to a Babel Block. Example:
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)" :post "my-babel-block(tbl=*this*)"
...
#+END:
#+end_example
#+begin_example
,#+name: my-babel-block
,#+begin_src elisp :var tbl=""
(append tbl
'(hline (Sunday "0.0")))
,#+end_src
#+end_example
*Beware!* You may want to add =:colnames t= to your Babel block. Otherwise
the table's header will be lost.
** Grand total
:PROPERTIES:
:CUSTOM_ID: grand-total
:END:
She (the user) may be tempted to add the grand total at the bottom of
the aggregation. Example of such a temptation:
#+begin_example
,#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
|-----------+--------------+----------------|
| Total | | 218 |
,#+TBLFM: @7$3=vsum(@I..@II)
,#+end
#+end_example
With OrgAggreagate post-processing, it is easy.
1. Just put in place a small algorithm to add two lines.
#+begin_example
:post (append *this* '(hline ("Total" "" "")))
▲ ▲ ▲
╰─────╮ │ │
the aggregated table╶─╯ │ │
one horizontal line╶─────╯ │
an empty cell to recieve the total╶──────╯
#+end_example
The =*this*= Lisp variable contains the aggregated table, just while the
post-processing takes place. The =append= Lisp function adds two rows to
the aggregated table, and returns the amended table.
Note that the =:post= parameter may be:
- a small Lisp expression, as in this example,
- a lambda expression, which parameter is the aggregated table,
- the name of a Lisp function, which is passed the aggregate table,
- the name of a Babel block, written in any supported language.
2. Fill the additional cell with a formula for the total.
#+begin_example
@>$2=vsum(@I..@II)
▲ ▲ ▲ ▲ ▲
│ │ │ │ ╰──────────────╮
│ │ │ ╰────────────────╮ │
│ │ ╰─────────────────╮ │ │
│ ╰────────────╮ │ │ │
╰──────────╮ │ │ │ │
last line╶─╯ │ │ │ │
second column╶─╯ │ │ │
sum all values between╶─╯ │ │
the first horizontal line╶─╯ │
and the second one╶──────────╯
#+end_example
In this way, the grand-total will be recomputed each time the
aggregation is refreshed (=C-c C-c=). Note the use of =@>$2= for the
coordinates of the cell receiving the total, instead of, for instance
=@7$3=. This ensures that the formula will continue to be applied on the
last row, even if the aggregated table grows later on. The same idea
applies for the =@I..@II= range, instead of, for instance =@2..@6=.
Even though OrgAggregate offers the user a versatil post-processing to
add a grand total, there are many reasons not to. If she does, she is
quietly entering the same nightmare which plagues spreadsheets.
Everything will become harder and harder to maintain.
It seems natural to add a grand total right below the column. But
suppose that now she also want the standard deviation of this same
column. Where to put it? There is a blank cell just left of the grand
total. She puts the standard deviation there:
#+begin_example
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
|-----------+--------------+----------------|
| Total | 27.409852 | 218 |
#+TBLFM: @7$3=vsum(@I..@II)::@7$2=vsdev(@I$3..@II$3)
#+end
#+end_example
But then this =27.409852= looks like the grand total of the second
column, but it is not.
Later on, she may want to further process this aggregated table, for
example to plot it. The grand total row will be an annoyance. It will
be tedious to get ride of it.
She should instead consider creating a second aggregation with the
grand total:
#+begin_example
,#+begin: aggregate :table original :cols "vsdev(Level) vsum(Quantity)"
| vsdev(Level) | vsum(Quantity) |
|--------------+----------------|
| 27.409852 | 218 |
,#+end
#+end_example
Easy, maintainable, no awkward decisions to remember and document.
She keeps it simple.
** Chaining
:PROPERTIES:
:CUSTOM_ID: chaining
:END:
The result of an aggregation may become the source of further
processing. To do that, just add a =#+NAME:= or =#+TBLNAME:= line
just above the aggregated table. Here is an example of a double
aggregation:
#+begin_example
#+NAME: squantity
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)"
| Day | SQuantity |
|-----------+-----------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
#+TBLFM: @1$2=SQuantity
#+END:
#+BEGIN: aggregate :table "squantity" :cols "vsum(SQuantity)"
| vsum(SQuantity) |
|-----------------|
| 218 |
#+END:
#+end_example
Note the spreadsheet cell formula =@1$2=SQuantity=, which changes the
column heading from it default =vsum(Quantity)= to =SQuantity=. This
new heading will survive any refresh.
Sometimes the name of the aggregated table is not found by some babel
block referencing it (Gnuplot blocks are among them). To fix that,
just exchange the =#+NAME:= and =#+BEGIN:= lines:
#+begin_example
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)"
#+NAME: squantity
| Day | SQuantity |
|-----------+-----------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
#+TBLFM: @1$2=SQuantity
#+END:
#+end_example
The =#.NAME:= line will survive when recomputing the aggregation (as
=#.TBLFM:= line survives)
* Pull & Push
:PROPERTIES:
:CUSTOM_ID: pull--push
:END:
Two modes are available: /pull/ & /push/.
** Pull mode
:PROPERTIES:
:CUSTOM_ID: pull-mode
:END:
In the /pull/ mode, we use so called /"dynamic blocks"/.
The resulting table knows how to build itself.
Example:
We have a source table which is unaware that it will be derived in an
aggregated table:
#+begin_example
#+NAME: source1
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
#+end_example
We create somewhere else a /dynamic block/ which carries the
specification of the aggregation:
#+begin_example
#+BEGIN: aggregate :table "source1" :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+END
#+end_example
Typing =C-c C-c= in the dynamic block recomputes it freshly.
** Push mode
:PROPERTIES:
:CUSTOM_ID: push-mode
:END:
In /push/ mode, the source table drives the creation of derived
tables. We specify the wanted results in =#+ORGTBL: SEND= directives
(as many as desired):
#+begin_example
#+ORGTBL: SEND derived1 orgtbl-to-aggregated-table :cols "vmean(Level) vsum(Quantity)"
#+ORGTBL: SEND derived2 orgtbl-to-aggregated-table :cols "Day vmean(Level) vsum(Quantity)"
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
#+end_example
We must create the receiving blocks somewhere else in the same file:
#+begin_example
#+BEGIN RECEIVE ORGTBL derived1
#+END RECEIVE ORGTBL derived1
#+end_example
#+begin_example
#+BEGIN RECEIVE ORGTBL derived2
#+END RECEIVE ORGTBL derived2
#+end_example
Then we come back to the source table and type =C-c C-c= with the
cursor on the 1st pipe of the table, to refresh the derived tables:
#+begin_example
#+BEGIN RECEIVE ORGTBL derived1
| vmean(Level) | vsum(Quantity) |
|---------------+----------------|
| 27.9285714286 | 218 |
#+END RECEIVE ORGTBL derived1
#+end_example
#+begin_example
#+BEGIN RECEIVE ORGTBL derived2
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+END RECEIVE ORGTBL derived2
#+end_example
** Pull or push ?
:PROPERTIES:
:CUSTOM_ID: pull-or-push-
:END:
Pull & push modes use the same engine in the background.
Thus, using either is just a matter of convenience.
Pull mode is the most straightforward. Also the wizard operates on the
pull mode only. Almost all examples in this documentation are in pull
mode. If you cannot decide, just use the pull mode.
_Glitch:_ in push mode you may see strange output like =\_{}=.
This is an escape generated by Org Mode (nothing to do with OrgAggregate).
It happens for the following characters: =&%#_^=
To disable that, in the =#+ORGTBL: SEND= line, add this parameter:
=:no-escape true=
* Debugging
:PROPERTIES:
:CUSTOM_ID: debugging
:END:
The work of OrgAggregate is to hand out pieces of the input table to
Calc (the Emacs calculator).
Is some intricate cases, it may be useful to see what is going on. The
debugging formatters come handy.
** Seeing the $ forms
:PROPERTIES:
:CUSTOM_ID: seeing-the--forms
:END:
Here is an example input table:
#+begin_example
#+name: inputdebug
| nn | aa |
|------+----|
| 1.23 | a |
| 7.65 | b |
| 8.46 | c |
|------+----|
| 2.44 | d |
| 6.81 | e |
#+end_example
And here is an aggregation to debug:
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10) vsum(aa+7)"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-------------+----------------|
| 0 | 173.4 | a + b + c + 21 |
| 1 | 92.5 | d + e + 14 |
#+END:
#+end_example
So far so good. But we would like to know what Calc did. To do so let
us add the =c= formatter.
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);c vsum(aa+7);c"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-------------+------------|
| 0 | vsum($1*10) | vsum($2+7) |
| 1 | vsum($1*10) | vsum($2+7) |
#+END:
#+end_example
Each output cell now contains the formula, with column names replaced
by dollar equivalent forms. But without any further processing.
** Seeing Calc formulas before evaluation
:PROPERTIES:
:CUSTOM_ID: seeing-calc-formulas-before-evaluation
:END:
Let us go one step further with the =C= formatter:
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);C vsum(aa+7);C"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-----------------------------+---------------------|
| 0 | vsum([1.23, 7.65, 8.46] 10) | vsum([a, b, c] + 7) |
| 1 | vsum([2.44, 6.81] 10) | vsum([d, e] + 7) |
#+END:
#+end_example
The dollar forms were replaced by Calc vectors made of input cells. No
foldings or simplifications went on. The vectors are slices of columns,
selected by OrgAggregate in response of the =hline= aggregation.
We see that multiplying by =10= or adding =7= is done on a Calc vector. It
happens that Calc knows how to multiply or add something to a
vector. OrgAggregate does not perform those operations, it delegates
them to Calc.
** Seeing Lisp internal form of Calc formulas
:PROPERTIES:
:CUSTOM_ID: seeing-lisp-internal-form-of-calc-formulas
:END:
We can also view the same results, formatted as Lisp forms (rather
than Calc forms) with the =Q= formatter:
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);Q vsum(aa+7);Q"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+---------------------------------------------------------------------------+-----------------------------------------------------------------------|
| 0 | (calcFunc-vsum (* (vec (float 123 -2) (float 765 -2) (float 846 -2)) 10)) | (calcFunc-vsum (+ (vec (var a var-a) (var b var-b) (var c var-c)) 7)) |
| 1 | (calcFunc-vsum (* (vec (float 244 -2) (float 681 -2)) 10)) | (calcFunc-vsum (+ (vec (var d var-d) (var e var-e)) 7)) |
#+END:
#+end_example
This is the internal, Lisp representation of Calc formulas.
** Example of debugging vsum(nn^2)
:PROPERTIES:
:CUSTOM_ID: example-of-debugging-vsumnn2
:END:
Beware of a formula like =vsum(nn^2)=. This gives the expected result,
although not in the obvious way. Let us see what happens, thanks to
the =C= debugging formatter:
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn^2);C"
| hline | vsum(nn^2) |
|-------+----------------------------|
| 0 | vsum([1.23, 7.65, 8.46]^2) |
| 1 | vsum([2.44, 6.81]^2) |
#+END:
#+end_example
We are not summing squares. We are squaring Calc vectors. Calc being a
mathematical tool, it interprets the product of two vectors as the sum
of the products element-wise, as a mathematician would do. Then =vsum=
is applied on a single resulting value. So =vsum= is useless in this
case. That can be confirmed:
#+begin_example
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn^2) nn^2 vprod(nn^2)"
| hline | vsum(nn^2) | nn^2 | vprod(nn^2) |
|-------+------------+---------+-------------|
| 0 | 131.607 | 131.607 | 131.607 |
| 1 | 52.3297 | 52.3297 | 52.3297 |
#+END:
#+end_example
Therefore, changing =vsum= to =vprod= does not change the result. This can
be unexpected.
** Summary of debugging formatters
:PROPERTIES:
:CUSTOM_ID: summary-of-debugging-formatters
:END:
To summarize the debugging settings:
- =c=: output Calc formula
- =C=: output Calc formula with dollar forms substituted by actual input data
- =q=: output Lisp formula
- =Q=: output Lisp formula with column forms substituted by actual input data
* Tricks
:PROPERTIES:
:CUSTOM_ID: tricks
:END:
This chapter collects some tricks that may be useful.
** Sorting
:PROPERTIES:
:CUSTOM_ID: sorting-0
:END:
#+begin_example
#+name: trick_table_1
| column |
|--------|
| 677 |
| 713 |
| 459 |
| 537 |
| 881 |
#+end_example
When several cells of a column need to be sorted, the Calc =calc-sort()= function is handy:
#+begin_example
#+BEGIN: aggregate :table "trick_table_1" :cols "(column) sort(column)"
| (column) | sort(column) |
|---------------------------+---------------------------|
| [677, 713, 459, 537, 881] | [459, 537, 677, 713, 881] |
#+END:
#+end_example
- =(column)= gives the list of values to aggregate, without aggregating them.
- =sort(column)= gives the same list sorted in ascending order.
** A few lowest or highest values
:PROPERTIES:
:CUSTOM_ID: a-few-lowest-or-highest-values
:END:
Used with =subvec()=, =sort()= can retrieve the two lowest or the two
highest values:
#+begin_example
#+BEGIN: aggregate :table "trick_table_1" :cols "subvec(sort(column),1,3) subvec(sort(column),count()-1)"
| subvec(sort(column),1,3) | subvec(sort(column),count()-1) |
|--------------------------+--------------------------------|
| [459, 537] | [713, 881] |
#+END:
#+end_example
- =subvec(...,1,3)= extracts the two first values: from =1= to =3= excluded.
- =subvec(...,count()-1)= extracts the two last values, numbered
=count()-1= and =count()=
And of course we may retrieve the average of the two first and the two
last values:
#+begin_example
#+BEGIN: aggregate :table "trick_table_1" :cols "vmean(subvec(sort(column),1,3)) vmean(subvec(sort(column),count()-1))"
| vmean(subvec(sort(column),1,3)) | vmean(subvec(sort(column),count()-1)) |
|---------------------------------+---------------------------------------|
| 498 | 797 |
#+END:
#+end_example
** Span of values
:PROPERTIES:
:CUSTOM_ID: span-of-values
:END:
=vmin()= and =vmax()= can compute the span of aggregated values:
#+begin_example
#+BEGIN: aggregate :table "trick_table_1" :cols "vmin(column) vmax(column) vmax(column)-vmin(column)"
| vmin(column) | vmax(column) | vmax(column)-vmin(column) |
|--------------+--------------+---------------------------|
| 459 | 881 | 422 |
#+END:
#+end_example
** No aggregation
:PROPERTIES:
:CUSTOM_ID: no-aggregation
:END:
Why would one want to use OrgAggregate while not aggregating? To
benefit from the other features of OrgAggregate:
- column rearrangement
- sorting
- formatting
- =#+TBLFM= survival
- row filtering
- preprocess
- postprocess
To do so, mention the virtual column =@#= in =:cols= and make it invisible
with =;<>=. As =@#= is different for each row, the aggregation will
consider each row as a separate group. Therefore, no aggregation on
another column will do anything more.
For example, here we:
- put =Color= as the first column (it is the second in the input),
- ignore the =Day= column,
- sort by =Level=,
- compute =Quantity/7=,
- format it with 2 digits after dot.
#+begin_example
#+BEGIN: aggregate :table "original" :cols "@#;<> Color Level;^n vmax(Quantity/7);'Q10';f2"
| Color | Level | Q10 |
|-------+-------+------|
| Blue | 6 | 1.14 |
| Blue | 7 | 0.71 |
| Blue | 11 | 1.29 |
| Blue | 12 | 2.29 |
| Blue | 15 | 2.14 |
| Blue | 25 | 0.43 |
| Red | 27 | 3.29 |
| Red | 30 | 1.57 |
| Blue | 33 | 2.57 |
| Red | 39 | 3.43 |
| Red | 41 | 4.14 |
| Red | 45 | 2.14 |
| Red | 49 | 4.29 |
| Red | 51 | 1.71 |
#+END:
#+end_example
We used the =vmax()= aggregating function on =Quantity/7=, because
otherwise we would get a vector with a single value. As there is a
single value, any aggregating function will do the trick: =vmin()=,
=head()=, =rtail()=, =vsum()=, =vprod()=, =vmean()=, =vgmean()=, =vhmean()=, =vspan()=,
=vmedian()=.
* Installation
:PROPERTIES:
:CUSTOM_ID: installation
:END:
Emacs package on Melpa: add the following lines to your =.emacs= file,
and reload it.
#+begin_example
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/") t)
(package-initialize)
#+end_example
You may also customize this variable:
#+begin_example
M-x customize-variable package-archives
#+end_example
Then browse the list of available packages and install =orgtbl-aggregate=
#+begin_example
M-x package-list-packages
#+end_example
Alternatively, you can download the lisp file, and load it:
#+begin_example
(load-file "orgtbl-aggregate.el")
#+end_example
* Authors, contributors
:PROPERTIES:
:CUSTOM_ID: authors-contributors
:END:
Authors
- Thierry Banel, tbanelwebmin at free dot fr, inception & implementation.
- Michael Brand, Calc unleashed, =#+TBLFM= survival, empty input cells, formatters.
Contributors
- Eric Abrahamsen, non-ASCII column names
- Alejandro Erickson, quoting non alphanumeric column names
- Uwe Brauer, simpler example in documentation, take
org-calc-default-modes preferences into account
- Peking Duck, fixed obsolete letf function
- Bill Hunker, discovered =\_{}= escape
- Dirk Schmitt, surviving =#.NAME:= line
- Dale Sedivec, case insensitive =#+NAME:= tags
- falloutphil, underscore in column names
- Baudilio Tejerina, t, T, U formatters
- Marco Pas, bug comparing empty string
- wuqui, sorting output table, filtering only
- Nicolas Viviani, output hlines
- Nils Lehmann, support old versions of the rx library
- Shankar Rao, =:post= post-processing
- Misohena (https://misohena.jp/blog/author/misohena),
double width Japanese characters (string-width vs. length)
- Kevin Brubeck Unhammer, ignore formatting cookies
- Tilmann Singer, more flexibility in duration format
- Piotr Panasiuk, =#+CAPTION:= and any tags survive
- Luis Miguel Hernanz, fix regex bug
- Jason Hemann, output column names no longer have quotes
- Tilmann Singer, computed aggregating bins, ="month(Date)"= in his use
case
* Changes
:PROPERTIES:
:CUSTOM_ID: changes
:END:
Top: earliest change. Bottom: latest change.
- Wizard now correctly asks for columns with =$1, $2...= names
when table header is missing
- Handle tables beginning with hlines
- Handle non-ASCII column names
- =:formula= parameter and =#+TBLFM= survival
- Empty cells are ignored.
- Empty output upon too small input set
- Fix ordering of output values
- Aggregations formulas may now be arbitrary expressions
- Table headers (and the lack of) are better handled
- Modifiers and formatters can now be specified as in the spreadsheet
- Aggregation function names can optionally have a leading =v=, like =sum= & =vsum=
- Increased performance on large data sets
- Tables can be named with =#+NAME:= besides =#+TBLNAME:=
- Document Melpa installation
- Support quoting of column names, like "a.b" or 'c/d'
- Disable =\_{}= escape
- =#+NAME:= inside =#+BEGIN:= survives
- Missing input cells handled as empty ones
- Back-port Org Mode =9.4= speed up
- Increase performance when inserting result into the buffer
- Aligned output in push mode
- Added a hash-table to speedup aggregation
- Back-port org-table-to-lisp which is now much faster
- =vlist(X)= now yields input cells verbatim were =(X)= yields Calc processed input cells
- Document dates handling and the =date()= function
- Implement =HH:MM:SS= durations and =T=, =t=, =U= formatters
- Sort output
- Create hlines in the output
- Missing :cond parameter means all columns
- Remove =C-c C-x i=, use standard =C-c C-x x= instead
- Avoid name collision between Calc functions and columns
- More readable & faster code
- Support for old versions of the rx library
- =:post= post-processing
- Propagate multiple rows source header to the aggregated header
- Ignore data rows containing formatting cookies
- Follow Org Mode way of handling Calc settings in Lisp code
- Hours in durations are no longer restricted to 2 digits
- 3x speedup =org-table-to-lisp= and avoid Emacs 27 to 30 incompatibilities
- =#+CAPTION:= and any other tag survive inside =#+BEGIN:=
- Output column names are now stripped from quotes, better reflecting
input names.
- Table-of-contents in README.org (thanks org-make-toc)
- Add formatters =c= =C= =q= =Q= (useful for debugging or understanding
OrgAggregate)
- Formulas involving =hline= like =vmean(hline*10)= are now taken into
account
- Documentation is now integrated right into Emacs in the =info= format.
Type =M-: (info "orgtbl-aggregate")=
- Input table may now be the result of a Babel script (virtual table).
- Better handling of user errors in the =:post= directive.
- Speedup of resulting table recalculation when there are formulas in
=#+tblfm:= or in =:formula=. The overall aggregation may be up to x6
faster and ÷5 less memory hungry.
- Circumvent an Org Mode bug in case there are a column-formula along
with a cell-formula, the cell-one not being calculated. (Bonus: 15%
speedup).
- Fix issue #24: bug in date parsing.
- Virtual pre-computed input columns.
- Better explanation of the input table reference syntax, including
distant tables and virtual table produced by Babel blocks.
- Support for CSV and JSON formatted input tables.
- New =@#= virtual column giving the number of each row, pretty much
like the Org table spreadsheet =@#= virtual column.
- Header and column names can be specified for CSV input tables, as
well as horizontal separators (=hline=).
- Aggregation of the titles of this README.
- New free-form wizard.
- Illustrate README with Uniline graphics.
- JSON and CSV input tables can now live inside Org Mode blocks.
- Example for computing a refreshable grand total.
- Document intervals and error-forms handling.
- Special columns =@#= and =hline= are handled by transpose.
* GPL 3 License
:PROPERTIES:
:CUSTOM_ID: gpl-3-license
:END:
Copyright (C) 2013-2026 Thierry Banel
orgtbl-aggregate is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
orgtbl-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
================================================
FILE: orgtbl-aggregate.el
================================================
;;; orgtbl-aggregate.el --- Aggregate an Org Mode table | + | + | into another table -*- coding:utf-8; lexical-binding: t;-*-
;; Copyright (C) 2013-2026 Thierry Banel
;; Authors:
;; Thierry Banel tbanelwebmin at free dot fr
;; Michael Brand michael dot ch dot brand at gmail dot com
;; Contributors:
;; Eric Abrahamsen, Alejandro Erickson Uwe Brauer, Peking Duck, Bill
;; Hunker, Dirk Schmitt, Dale Sedivec, falloutphil, Baudilio
;; Tejerina, Marco Pas, wuqui, Nicolas Viviani, Nils Lehmann,
;; Shankar Rao, Misohena, Kevin Brubeck Unhammer, Tilmann Singer,
;; Piotr Panasiuk, Luis Miguel Hernanz, Jason Hemann
;; Package-Requires: ((emacs "26.1"))
;; Version: 1.0
;; Keywords: data, extensions
;; URL: https://github.com/tbanel/orgaggregate/blob/master/README.org
;; orgtbl-aggregate is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;; orgtbl-aggregate is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see .
;;; Commentary:
;;
;; A new org-mode table is automatically updated,
;; based on another table acting as a data source
;; and user-given specifications for how to perform aggregation.
;;
;; Example:
;; Starting from a source table of activities and quantities
;; (whatever they are) over several days,
;;
;; #+TBLNAME: original
;; | Day | Color | Level | Quantity |
;; |-----------+-------+-------+----------|
;; | Monday | Red | 30 | 11 |
;; | Monday | Blue | 25 | 3 |
;; | Tuesday | Red | 51 | 12 |
;; | Tuesday | Red | 45 | 15 |
;; | Tuesday | Blue | 33 | 18 |
;; | Wednesday | Red | 27 | 23 |
;; | Wednesday | Blue | 12 | 16 |
;; | Wednesday | Blue | 15 | 15 |
;; | Thursday | Red | 39 | 24 |
;; | Thursday | Red | 41 | 29 |
;; | Thursday | Red | 49 | 30 |
;; | Friday | Blue | 7 | 5 |
;; | Friday | Blue | 6 | 8 |
;; | Friday | Blue | 11 | 9 |
;;
;; an aggregation is built for each day (because several rows
;; exist for each day), typing C-c C-c
;;
;; #+BEGIN: aggregate :table original :cols "Day mean(Level) sum(Quantity)"
;; | Day | mean(Level) | sum(Quantity) |
;; |-----------+-------------+---------------|
;; | Monday | 27.5 | 14 |
;; | Tuesday | 43 | 45 |
;; | Wednesday | 18 | 54 |
;; | Thursday | 43 | 83 |
;; | Friday | 8 | 22 |
;; #+END
;;
;; A wizard can be used:
;; C-c C-x x aggregate
;;
;; Full documentation here:
;; https://github.com/tbanel/orgaggregate/blob/master/README.org
;;; Requires:
(require 'calc-ext)
(require 'calc-aent)
(require 'calc-alg)
(require 'calc-arith)
(require 'org)
(require 'org-table)
(require 'org-id)
(require 'thingatpt) ;; just for thing-at-point--read-from-whole-string
(eval-when-compile (require 'cl-lib))
(require 'rx)
(require 'json)
(eval-when-compile
(cl-proclaim '(optimize (speed 3) (safety 0))))
;;; Code:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The venerable Calc is used thoroughly by the Aggregate package.
;; A few bugs were found.
;; They have been fixed in recent versions of Emacs
;; Uncomment the fixes if needed
;(defun math-max-list (a b)
; (if b
; (if (or (Math-anglep (car b)) (eq (caar b) 'date)
; (and (eq (car (car b)) 'intv) (math-intv-constp (car b)))
; (math-infinitep (car b)))
; (math-max-list (math-max a (car b)) (cdr b))
; (math-reject-arg (car b) 'anglep))
; a))
;
;(defun math-min-list (a b)
; (if b
; (if (or (Math-anglep (car b)) (eq (caar b) 'date)
; (and (eq (car (car b)) 'intv) (math-intv-constp (car b)))
; (math-infinitep (car b)))
; (math-min-list (math-min a (car b)) (cdr b))
; (math-reject-arg (car b) 'anglep))
; a))
;; End of Calc fixes
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; creating long lists in the right order may be done
;; - by (nconc) but behavior is quadratic
;; - by (cons) (nreverse)
;; a third way involves keeping track of the last cons of the growing list
;; a cons at the head of the list is used for housekeeping
;; the actual list is (cdr ls)
;;
;; A list with 4 elements:
;; ╭─┬─╮ ╭────┬─╮ ╭────┬─╮ ╭────┬─╮ ╭────┬─╮
;; │◦│◦┼▶┤val1│◦┼▶┤val2│◦┼▶┤val3│◦┼▶┤val4│◦┼▶╴nil
;; ╰┼┴─╯ ╰────┴─╯ ╰────┴─╯ ╰────┴─╯ ╰─┬──┴─╯
;; │ ▲
;; ╰─────────────────────────────────╯
;;
;; A newly created, empty list
;; ╭─┬─╮
;; │◦│◦┼▶─nil
;; ╰┼┴┬╯
;; │ ▲
;; ╰─╯
(eval-when-compile
(defmacro orgtbl-aggregate--list-create ()
"Create an appendable list."
`(let ((x (cons nil nil)))
(setcar x x)))
(defmacro orgtbl-aggregate--list-append (ls value)
"Append VALUE at the end of LS in O(1) time."
`(setcar ,ls (setcdr (car ,ls) (cons ,value nil))))
(defmacro orgtbl-aggregate--list-get (ls)
"Return the regular Lisp list from LS."
`(cdr ,ls))
(defmacro orgtbl-aggregate--pop-simple (place)
"Like (pop PLACE), but without returning (car PLACE)."
`(setq ,place (cdr ,place)))
(defmacro orgtbl-aggregate--pop-leading-hline (table)
"Remove leading hlines from TABLE, if any."
`(while (not (listp (car ,table)))
(orgtbl-aggregate--pop-simple ,table)))
(defmacro orgtbl-aggregate--string-match-p (regexp string &optional start)
"Same as standard `string-match-p'
but written as a defmacro instead of a defsubst,
which saves 4 or 5 byte-codes at each call."
`(string-match ,regexp ,string ,start t))
)
(defun orgtbl-aggregate--alist-get-remove (key alist)
"A variant of alist-get which removes an entry once read.
ALIST is a list of pairs (key . value).
Search ALIST for a KEY. If found, replace the key in (key . value)
by nil, and return value. If nothing is found, return nil."
(let ((x (assq key alist)))
(when x
(setcar x nil)
(cdr x))))
(defun orgtbl-aggregate--plist-get-remove (params prop)
"Like `plist-get', but also remove PROP from PARAMS."
(let ((v (plist-get params prop)))
(if v
(setcar (memq prop params) nil))
v))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The function (org-table-to-lisp) have been greatly enhanced
;; in Org Mode version 9.4
;; To benefit from this speedup in older versions of Org Mode,
;; this function is copied here with a slightly different name
;; It has also undergone near 3x speedup,
;; - by not using regexps
;; - achieving the shortest bytecode
;; Furthermore, this version avoids the
;; inhibit-changing-match-data and looking-at
;; incompatibilities between Emacs-27 and Emacs-30
(defun orgtbl-aggregate--table-to-lisp (&optional txt)
"Convert the table at point to a Lisp structure.
The structure will be a list. Each item is either the symbol `hline'
for a horizontal separator line, or a list of field values as strings.
The table is taken from the parameter TXT, or from the buffer at point."
(if txt
(with-temp-buffer
(buffer-disable-undo)
(insert txt)
(goto-char (point-min))
(orgtbl-aggregate--table-to-lisp))
(save-excursion
(goto-char (org-table-begin))
(let (table)
(while (progn (skip-chars-forward " \t")
(eq (following-char) ?|))
(forward-char)
(push
(if (eq (following-char) ?-)
'hline
(let (row)
(while (progn (skip-chars-forward " \t")
(not (eolp)))
(let ((q (point)))
(skip-chars-forward "^|\n")
(goto-char
(let ((p (point)))
(unless (eolp) (setq p (1+ p)))
(skip-chars-backward " \t" q)
(push
(buffer-substring-no-properties q (point))
row)
p))))
(nreverse row)))
table)
(forward-line))
(nreverse table)))))
;; There is no CSV parser bundled with Emacs. In order to avoid a
;; dependency on a package, here is an implementation of a parser. It
;; is made of the same technology as `orgtbl-aggregate--table-to-lisp'
;; (which is now integrated into the newest versions of Emacs). It is
;; probably as fast as can be in Emacs-Lisp byte-code.
(defun orgtbl-aggregate--csv-to-lisp (header colnames)
"Convert current buffer in CSV to Lisp.
It recognize cells protected by double quotes, and cells not protected.
When a cell is not protected, blanks are kept.
When a cell is protected, blanks before the first double quote are ignored.
Double double quotes are recognized within a cell double-quoted.
The last line may or may not end in a newline.
Separators are comma, semicolon, or TAB. They can be mixed.
If a row is empty, it is considered as a separator, and translated
to `hline', the Org table horizontal separator.
HEADER non nil means that the first row must be interpreted as a header.
COLNAMES, if not nil, is a list of column names."
(goto-char (point-min))
(let (table)
(while (not (eobp))
(let (row)
(while (not (eolp))
(let ((p (point)))
(skip-chars-forward " ")
(if (eq (following-char) ?\")
(let (dquote)
(forward-char 1)
(setq p (point))
(while
(progn
(skip-chars-forward "^\"")
(forward-char 1)
(if (eq (following-char) ?\")
(progn (forward-char 1)
(setq dquote t)))))
(push
(let ((cell
(buffer-substring-no-properties p (1- (point)))))
(if dquote
(string-replace "\"\"" "\"" cell)
cell))
row)
(skip-chars-forward " "))
(skip-chars-forward "^,;\t\n")
(push
(buffer-substring-no-properties p (point))
row))
(skip-chars-forward ",;\t" (1+ (point)))))
(push
(if row (nreverse row) 'hline)
table)
(or (eobp) (forward-char 1))))
(setq table (nreverse table))
(if header
(setcdr table (cons 'hline (cdr table))))
(if colnames
(setq table (cons colnames (cons 'hline table))))
table))
;; A few rx abbreviations
;; each time a bit of a regexp is used twice or more,
;; it makes sense to define an abbrev
(eval-when-compile ;; not used at runtime
;; search for table name, such as:
;; #+tablename: mytable
(rx-define tblname
(seq bol (* blank) "#+" (? "tbl") "name:" (* blank)))
;; skip lines beginning with # in order to reach the start of table
(rx-define skip-meta-table (firstchars)
(seq
(0+ (0+ blank) (? firstchars (0+ nonl)) "\n")
(0+ blank) "|"))
;; just to get ride of a few parenthesis
(rx-define notany (&rest list)
(not (any list)))
;; match quoted column names, like
;; 'col a' "col b" colc
(rx-define quotedcolname (&rest bare)
(or
(seq ?' (* (notany ?' )) ?' )
(seq ?\" (* (notany ?\")) ?\")
bare))
;; match a column name not protected by quotes
(rx-define nakedname
(+ (any "$._#@" word)))
)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Here is a bunch of useful utilities,
;; generic enough to be detached from the orgtbl-aggregate package.
;; For the time being, they are here.
(defun orgtbl-aggregate--list-local-tables (file)
"Search for available tables in FILE.
If FILE is nil, use current buffer."
(interactive)
(with-current-buffer
(if file (find-file-noselect file) (current-buffer))
(save-excursion
(goto-char (point-min))
(let ((case-fold-search t))
(cl-loop
while
(re-search-forward
(rx tblname (group (*? nonl)) (* blank) eol)
nil t)
collect (match-string-no-properties 1))))))
(defun orgtbl-aggregate--table-from-babel (name-or-id)
"Retrieve an input table as the result of running a Babel block.
NAME-OR-ID is the usual Org convention for pointing to a distant reference.
Examples: babel, file:babel, file:babel[1:3,2:5], file:babel(p1=…,p2=…)
This function could work also for a table,
but this has already been short-circuited."
;; A user error is generated in case no Babel block is found
(let ((table (org-babel-ref-resolve name-or-id)))
(and
table
(consp table)
(or (eq (car table) 'hline)
(consp (car table)))
table)))
(defun orgtbl-aggregate--block-from-name (file name)
"Parse an Org table named NAME in a distant Org file named FILE.
FILE is a filename with possible relative or absolute path.
If FILE is nil, look in the current buffer."
(with-current-buffer
(if file
(find-file-noselect file)
(current-buffer))
(save-excursion
(goto-char (point-min))
(let ((case-fold-search t))
(if (re-search-forward
(rx ;; a single regexp :)
tblname (literal name) (* blank) "\n"
(0+ blank) "#+begin" (0+ nonl) "\n"
(group (*? (or any "\n")))
bol (* space) "#+end")
nil t)
(match-string-no-properties 1))))))
(defun orgtbl-aggregate--table-from-csv (file name params)
"Parse a CSV formatted table located in FILE.
If NAME is nil, then FILE is supposed to contain just one CSV table.
If NAME is given, it is supposed to be an Org block name which contains
a CSV table.
The cell-separator is currently guessed.
Currently, there is no header."
(let ((header) (colnames) (block))
(cl-loop
for p on (cdr (read params))
do
(cond
((eq (car p) 'header)
(setq header t))
((eq (car p) 'colnames)
(setq p (cdr p))
(setq colnames (car p)))
(t
(message "parameter %S not recognized" (car p)))))
(if name
(setq block (orgtbl-aggregate--block-from-name file name)))
(with-temp-buffer
(if name
(insert block)
(insert-file-contents file))
(orgtbl-aggregate--csv-to-lisp header colnames))))
(defun orgtbl-aggregate--table-from-json (file name _params)
"Parse a JSON formatted table located in FILE.
FILE is a filename with possible relative or absolute path.
Currently, the accepted format is
[[\"COL1\",\"COL2\",…]
\"hline\"
[\"VAL1\",\"VAL2\",…]
{\"key1\":\"val1\", \"key2\":\"val2\",…},
…
]
Numbers do not need to be quoted.
Horizontal lines may be: \"hline\", null, [], {}.
A mixture of vector and hash-objects is allowed.
Therfore the styles vector-of-vectors and vector-of-hash-objects
are supported.
A header containing the column names may be given as the first row,
(which must be a vector) followed by an horizontal line.
Keys not found in the header (or if there is no header), are added
to the column names."
(let ((json-object-type 'alist)
(json-array-type 'list)
(json-key-type 'string))
(let ((json
(if name
(json-read-from-string (orgtbl-aggregate--block-from-name file name))
(json-read-file file)))
(colnames ())
(result))
(when (and (cddr json) ;; at least 2 rows
(consp (car json)) ;; first row is a vector
(not (consp (cadr json)))) ;; second row is an hline
(setq colnames (car json)) ;; then first row contains column names
(setq json (cddr json)))
(setq
result
(cl-loop
for row in json
if (not row) ;; [], {}, null
collect 'hline ;; are 'hline
else if (stringp row) ;; "symbol"
collect (intern row) ;; becomes 'symbol
else if (and (consp row) (consp (car row)))
collect ;; case of an hash-object
(progn
(cl-loop
for icell in row
if (and (consp icell) (not (member (car icell) colnames)))
do (setq colnames `(,@colnames ,(car icell))))
(let ((vec (make-list (length colnames) nil)))
(cl-loop
for icell in row
do (cl-loop
for colname in colnames
for ocell on vec
if (equal colname (car icell))
do (setcar ocell (cdr icell))))
vec))
else ;; case of a vector
collect row))
(if colnames
`(,colnames hline ,@result)
result))))
(defun orgtbl-aggregate--table-from-name (file name)
"Parse an Org table named NAME in a distant Org file named FILE.
FILE is a filename with possible relative or absolute path.
If FILE is nil, look in the current buffer."
(with-current-buffer
(if file
(find-file-noselect file)
(current-buffer))
(save-excursion
(goto-char (point-min))
(and
(let ((case-fold-search t))
(re-search-forward
(rx
tblname (literal name) (* blank) eol
(skip-meta-table "#"))
nil t))
(orgtbl-aggregate--table-to-lisp)))))
(defun orgtbl-aggregate--table-from-id (id)
"Parse a table following a header in a distant Org file.
The header have an ID property equal to ID in a PROPERTY drawer."
(let ((id-loc (org-id-find id 'marker)))
(when (and id-loc (markerp id-loc))
(with-current-buffer (marker-buffer id-loc)
(save-excursion
(goto-char (marker-position id-loc))
(move-marker id-loc nil)
(and
(let ((case-fold-search t))
(re-search-forward
(rx point (skip-meta-table (any "*#:")))
nil t))
(orgtbl-aggregate--table-to-lisp)))))))
(defun orgtbl-aggregate--nil-if-empty (field)
(and
field
(not (string-match-p (rx bos (* blank) eos) field))
field))
(defun orgtbl-aggregate--parse-locator (locator)
"Parse LOCATOR, a description of where to find the input table.
The result is a vector containing:
[
FILE ; optional file where the table/Babel/CSV/JSON may be found
NAME ; name of table/Babel denoted by #+name:
ORGID ; Org Mode id in a property drawer (exclusive with file+name)
PARAMS ; optional parameters to pass to babel/CSV/JSON
SLICE ; optional slicing of the resultin table, like [0:7]
]
If LOCATOR looks like NAME(params…)[slice] or just NAME, then NAME
is searched in the Org Mode database, and if found it is interpreted
as an Org Id and put in the `orgid' field."
(unless locator (setq locator ""))
(unless
(string-match
(rx
bos
(* space)
(? (group-n 1 (* (notany ":"))) ":")
(* space)
( group-n 2 (* (notany "[]():")))
(* space)
(? (group-n 3 "(" (* nonl) ")"))
(* space)
(? (group-n 4 "[" (* nonl) "]"))
(* space)
eos)
locator)
(user-error "Malformed table reference %S" locator))
(let ((file (orgtbl-aggregate--nil-if-empty (match-string 1 locator)))
(name (orgtbl-aggregate--nil-if-empty (match-string 2 locator)))
(orgid )
(params (orgtbl-aggregate--nil-if-empty (match-string 3 locator)))
(slice (orgtbl-aggregate--nil-if-empty (match-string 4 locator))))
(when (and
(not file)
(progn
(unless org-id-locations (org-id-locations-load))
(and org-id-locations
(hash-table-p org-id-locations)
(gethash name org-id-locations))))
(setq orgid name)
(setq name nil))
(vector file name orgid params slice)))
(defun orgtbl-aggregate--assemble-locator (file name orgid params slice)
"Assemble fields of a locator as a string.
FILE NAME ORGID PARAMS SLICE are the 5 fields composing a locator.
Many of them are optional.
The result is a locator suitable for orgtbl-aggregate and Org Mode."
(unless params (setq params ""))
(unless slice (setq slice ""))
(setq file (orgtbl-aggregate--nil-if-empty file ))
(setq orgid (orgtbl-aggregate--nil-if-empty orgid))
(cond
(orgid (format "%s%s%s" orgid params slice))
(file (format "%s:%s%s%s" file (or name "") params slice))
(t (format "%s%s%s" name params slice))))
(defun orgtbl-aggregate-table-from-any-ref (name-or-id)
"Find a table referenced by NAME-OR-ID.
The reference is all the accepted Org references,
and additionally pointers to CSV or JSON files.
The pointed to object may also be a Babel block, which when executed
returns an Org table. Parameters may be passed to the Babel block
in parenthesis.
A slicing may be applied to the table, to select rows or columns.
The syntax for slicing is like [1:3] or [1:3,2:5].
Return it as a Lisp list of lists.
An horizontal line is translated as the special symbol `hline'."
(unless (stringp name-or-id)
(setq name-or-id (format "%s" name-or-id)))
(let*
((struct (orgtbl-aggregate--parse-locator name-or-id))
(file (aref struct 0))
(name (aref struct 1))
(orgid (aref struct 2))
(params (aref struct 3))
(slice (aref struct 4))
(table
(cond
;; name-or-id = "file:(csv …)"
((and params (string-match-p (rx bos "(csv") params))
(orgtbl-aggregate--table-from-csv file name params))
;; name-or-id = "file:(json …)"
((and params (string-match-p (rx bos "(json") params))
(orgtbl-aggregate--table-from-json file name params))
;; name-or-id = "34cbc63a-c664-471e-a620-d654b26ffa31"
;; pointing to a header in a distant org file, followed by a table
(orgid
(orgtbl-aggregate--table-from-id orgid))
;; name-or-id = "babel(p=…)" or "file:babel(p=…)"
((and params
(orgtbl-aggregate--table-from-babel
(if file
(format "%s:%s%s" file name params)
(format "%s%s" name params)))))
;;name-or-id = "table" or "file:table"
((orgtbl-aggregate--table-from-name file name))
;; name-or-id = "babel" or "file:babel"
((orgtbl-aggregate--table-from-babel
(if file
(format "%s:%s" file name)
name)))
;; everything failed
(t
(user-error
"Cannot find table or babel block with reference %S"
name-or-id)))))
(if slice
(org-babel-ref-index-list slice table)
table)))
(defun orgtbl-aggregate--split-string-with-quotes (string)
"Like (split-string STRING), but with quote protection.
Single and double quotes protect space characters,
and also single quotes protect double quotes
and the other way around."
(let ((l (length string))
(start 0)
(result (orgtbl-aggregate--list-create)))
(save-match-data
(while (and (< start l)
(string-match
(rx
(* blank)
(group (+ (quotedcolname (notany " '\"")))))
string start))
(orgtbl-aggregate--list-append result (match-string 1 string))
(setq start (match-end 1))))
(orgtbl-aggregate--list-get result)))
(defun orgtbl-aggregate--merge-list-into-single-string (cols)
"Create a string representing the list COLS.
This is the opposite of `orgtbl-aggregate--split-string-with-quotes'."
(if (listp cols)
(mapconcat
(lambda (x)
(format "%s" x)) ;; x already a string? returns it unchanged
cols " ")
cols))
(defun orgtbl-aggregate--colname-to-int (colname table &optional err)
"Convert the COLNAME into an integer.
COLNAME is a column name of TABLE.
The first column is numbered 1.
COLNAME may be:
- a dollar form, like $5 which is converted to 5
- an alphanumeric name which appears in the column header (if any)
- the special symbol `hline' which is converted into 0
If COLNAME is quoted (single or double quotes),
quotes are removed beforhand.
When COLNAME does not match any actual column,
an error is generated if ERR optional parameter is true
otherwise nil is returned."
(if (symbolp colname)
(setq colname (symbol-name colname)))
(if (string-match
(rx
bos
(or
(seq ?' (group-n 1 (* (notany ?' ))) ?' )
(seq ?\" (group-n 1 (* (notany ?\"))) ?\"))
eos)
colname)
(setq colname (match-string 1 colname)))
;; skip first hlines if any
(orgtbl-aggregate--pop-leading-hline table)
(cond ((string= colname "")
(and err (user-error "Empty column name")))
((string= colname "@#")
0)
((string= colname "hline")
(1+ (length (car table))))
((string-match (rx bos "$" (group (+ digit)) eos) colname)
(let ((n (string-to-number (match-string 1 colname))))
(if (<= n (length (car table)))
n
(if err
(user-error "Column %s outside table" colname)))))
((and
(memq 'hline table)
(cl-loop
for h in (car table)
for i from 1
thereis (and (string= h colname) i))))
(err
(user-error "Column %s not found in table" colname))))
(defun orgtbl-aggregate--insert-make-spaces (n spaces-cache)
"Make a string of N spaces.
Caches results into SPACES-CACHE to avoid re-allocating
again and again the same string."
(if (< n (length spaces-cache))
(or (aref spaces-cache n)
(aset spaces-cache n (make-string n ? )))
(make-string n ? )))
;; Time optimization: surprisingly,
;; (insert (concat a b c)) is faster than
;; (insert a b c)
;; Therefore, we build the Org Mode representation of a table
;; as a list of strings which get concatenated into a huge string.
;; This is faster and less garbage-collector intensive than
;; inserting cells one at a time in a buffer.
;;
;; benches:
;; insert a large 3822 rows × 16 columns table
;; - one row at a time or as a whole
;; - with or without undo active
;; repeat 10 times
;;
;; with undo, one row at a time
;; (3.587732240 40 2.437140552)
;; (3.474445440 39 2.341087725)
;;
;; without undo, one row at a time
;; (3.127574093 33 2.001691096)
;; (3.238456106 33 2.089536034)
;;
;; with undo, single huge string
;; (3.030763545 30 1.842303196)
;; (3.012367879 30 1.841319998)
;;
;; without undo, single huge string
;; (2.499138596 21 1.419285666)
;; (2.403039955 21 1.338347655)
;; ▲ ▲ ▲
;; │ │ ╰──╴CPU time for GC
;; │ ╰─────────╴number of GC
;; ╰─────────────────╴overall CPU time
(defun orgtbl-aggregate--elisp-table-to-string (table)
"Convert TABLE to a string formatted as an Org Mode table.
TABLE is a list of lists of cells. The list may contain the
special symbol `hline' to mean an horizontal line."
(let* ((nbcols (cl-loop
for row in table
maximize (if (listp row) (length row) 0)))
(maxwidths (make-list nbcols 1))
(numbers (make-list nbcols 0))
(non-empty (make-list nbcols 0))
(spaces-cache (make-vector 100 nil)))
;; compute maxwidths
(cl-loop for row in table
do
(cl-loop for cell on row
for mx on maxwidths
for nu on numbers
for ne on non-empty
for cellnp = (car cell)
do (cond ((not cellnp)
(setcar cell (setq cellnp "")))
((not (stringp cellnp))
(setcar cell (setq cellnp (format "%s" cellnp)))))
if (string-match-p org-table-number-regexp cellnp)
do (setcar nu (1+ (car nu)))
unless (string= cellnp "")
do (setcar ne (1+ (car ne)))
if (< (car mx) (string-width cellnp))
do (setcar mx (string-width cellnp))))
;; change meaning of numbers from quantity of cells with numbers
;; to flags saying whether alignment should be left (number alignment)
(cl-loop for nu on numbers
for ne in non-empty
do
(setcar nu (< (car nu) (* org-table-number-fraction ne))))
;; create well padded and aligned cells
(let ((bits (orgtbl-aggregate--list-create)))
(cl-loop for row in table
do
(if (listp row)
(cl-loop for cell in row
for mx in maxwidths
for nu in numbers
for pad = (- mx (string-width cell))
do
(orgtbl-aggregate--list-append bits "| ")
(cond
;; no alignment
((<= pad 0)
(orgtbl-aggregate--list-append bits cell))
;; left alignment
(nu
(orgtbl-aggregate--list-append bits cell)
(orgtbl-aggregate--list-append
bits
(orgtbl-aggregate--insert-make-spaces pad spaces-cache)))
;; right alignment
(t
(orgtbl-aggregate--list-append
bits
(orgtbl-aggregate--insert-make-spaces pad spaces-cache))
(orgtbl-aggregate--list-append bits cell)))
(orgtbl-aggregate--list-append bits " "))
(cl-loop for bar = "|" then "+"
for mx in maxwidths
do
(orgtbl-aggregate--list-append bits bar)
(orgtbl-aggregate--list-append bits (make-string (+ mx 2) ?-))))
(orgtbl-aggregate--list-append bits "|\n"))
;; remove the last \n because Org Mode re-adds it
(setcar (car bits) "|")
(mapconcat #'identity (orgtbl-aggregate--list-get bits)))))
(defun orgtbl-aggregate--insert-elisp-table (table)
"Insert TABLE in current buffer at point.
TABLE is a list of lists of cells. The list may contain the
special symbol `hline' to mean an horizontal line."
;; inactivating jit-lock-after-change boosts performance a lot
(cl-letf (((symbol-function 'jit-lock-after-change) (lambda (_a _b _c)) ))
(insert (orgtbl-aggregate--elisp-table-to-string table))))
(defun orgtbl-aggregate--cell-to-string (cell)
"Convert CELL (a cell in the input table) to a string if it is not already."
(cond
((not cell) cell)
((stringp cell) cell)
((numberp cell) (number-to-string cell))
((symbolp cell) (symbol-name cell))
(t (error "cell %S is not a number neither a string" cell))))
(defun orgtbl-aggregate--get-header-table (table &optional asstring)
"Return the header of TABLE as a list of column names.
When ASSTRING is true, the result is a string which concatenates the
names of the columns. TABLE may be a Lisp list of rows, or the
name or id of a distant table. The function takes care of
possibly missing headers, and in this case returns a list
of $1, $2, $3... column names.
Actual column names which are not fully alphanumeric are quoted."
(unless (consp table)
(setq table
(condition-case _err
(orgtbl-aggregate-table-from-any-ref table)
(error
'(("$1" "$2" "$3" "…") hline)))))
(orgtbl-aggregate--pop-leading-hline table)
(let ((header
(if (memq 'hline table)
(cl-loop for x in (car table)
do (setq x (orgtbl-aggregate--cell-to-string x))
collect
(if (string-match-p
(rx bos nakedname eos)
x)
x
(format "\"%s\"" x)))
(cl-loop for _x in (car table)
for i from 1
collect (format "$%s" i)))))
(if asstring
(mapconcat #'identity header " ")
header)))
;; The *this* variable is accessible to the user.
;; It refers to the aggregated table before it is "printed"
;; into the buffer, so that it can be post-processed.
(defvar *this*)
(defun orgtbl-aggregate--post-process (table post)
"Post-process the aggregated TABLE according to the :post header.
POST might be:
- a reference to a babel-block, for example:
:post \"myprocessor(inputtable=*this*)\"
and somewhere else:
#+name: myprocessor
#+begin_src language :var inputtable=
...
#+end_src
- a Lisp lambda with one parameter, for example:
:post (lambda (table) (append table \\'(hline (\"total\" 123))))
- a Lisp function with one parameter, for example:
:post my-lisp-function
- a Lisp expression which will be evaluated
the *this* variable will contain the TABLE
In all those cases, the result must be a Lisp value compliant
with an Org Mode table."
(cond
((null post) table)
((functionp post)
(apply post table ()))
((stringp post)
(let ((*this* table))
(condition-case err
(org-babel-ref-resolve post)
(error
(message "error: %S" err)
(condition-case err2
(orgtbl-aggregate--post-process
table
(thing-at-point--read-from-whole-string post))
(error
(user-error
":post %S ends in an error
- as a Babel block: %s
- not a valid Lisp expression: %s"
post err err2)))))))
((listp post)
(let ((*this* table))
(eval post)))
(t (user-error ":post %S header could not be understood" post))))
(defun orgtbl-aggregate--recover-TBLFM (content)
"Return a line begining with #+tblfm: within CONTENT, if any."
(and
content
(let ((case-fold-search t))
(string-match
(rx bol (* blank) (group "#+tblfm:" (* nonl)))
content))
(match-string 1 content)))
(defun orgtbl-aggregate--recalculate-fast ()
"Wrapper arround `org-table-recalculate'.
The standard `org-table-recalculate' function is slow because
it must handle lots of cases. Here the table is freshely created,
therefore a lot of special handling and cache updates can be
safely bypassed. Moreover, the alignment of the resulting table
is delegated to orgtbl-aggregate, which is fast.
The result is a speedup up to x6, and a memory consumption
divided by up to 5. It makes a difference for large tables."
(let ((old (symbol-function 'org-table-goto-column)))
(cl-letf (((symbol-function 'org-fold-core--fix-folded-region)
(lambda (_a _b _c)))
((symbol-function 'jit-lock-after-change)
(lambda (_a _b _c)))
;; Warning: this org-table-goto-column trick fixes a bug
;; in org-table.el around line 3084, when computing
;; column-count. The bug prevents single-cell formulas
;; creating the cell in some rare cases.
((symbol-function 'org-table-goto-column)
(lambda (n &optional on-delim _force)
;; △
;;╭───────────────────────────╯
;;╰╴parameter is forcibly changed to t╶─╮
;; ╭───────────────╯
;; ▽
(funcall old n on-delim t))))
(condition-case nil
(org-table-recalculate t t)
;; △ △
;; for all lines╶────────╯ │
;; do not re-align╶────────╯
(args-out-of-range nil)))))
(defun orgtbl-aggregate--table-recalculate (content formula)
"Update the #+TBLFM: line and recompute all formulas.
The computed table may have formulas which need to be recomputed.
This function adds a #+TBLFM: line at the end of the table.
It merges old formulas (if any) contained in CONTENT,
with new formulas (if any) given in the `formula' directive."
(let ((tblfm (orgtbl-aggregate--recover-TBLFM content))) ;; recover a #+tblfm: line
(if (stringp formula)
;; There is a :formula directive. Add it if not already there
(if tblfm
(unless (string-match-p (regexp-quote formula) tblfm)
(setq tblfm (format "%s::%s" tblfm formula)))
(setq tblfm (format "#+TBLFM: %s" formula))))
(when tblfm
;; There are formulas. They need to be evaluated.
(end-of-line)
(insert "\n" tblfm)
(forward-line -1)
(orgtbl-aggregate--recalculate-fast)
;; Realign table after org-table-recalculate have changed or added
;; some cells. It is way faster to re-read and re-write the table
;; through orgtbl-aggregate routines than letting org-mode do the job.
(let* ((table (orgtbl-aggregate--table-to-lisp))
(width
(cl-loop for row in table
if (consp row)
maximize (length row))))
(cl-loop
for row in table
if (and (consp row) (< (length row) width))
do (nconc row (make-list (- width (length row)) nil)))
(delete-region (org-table-begin) (org-table-end))
(insert (orgtbl-aggregate--elisp-table-to-string table) "\n")))))
;; bazilo]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The Org Table Aggregation package really begins here
(defun orgtbl-aggregate--replace-colnames-nth (table expression)
"Replace occurrences of column names in Lisp EXPRESSION.
Replacements are forms like (nth N row),
N being the numbering of columns.
Doing so, EXPRESSION is ready to be computed against a TABLE row."
(cond
((listp expression)
(cons (car expression)
(cl-loop for x in (cdr expression)
collect
(orgtbl-aggregate--replace-colnames-nth table x))))
((numberp expression)
expression)
(t
(let ((n (orgtbl-aggregate--colname-to-int expression table)))
(if n
`(nth ,n orgtbl-aggregate--row)
expression)))))
(defun orgtbl-aggregate--to-frux (formula table involved)
"Parse FORMULA replacing column names with Frux(NN).
NN is the column position as it appears in TABLE.
Take into account protection of non-alphanumeric names
by single or double quotes.
Also replace sum, mean, etc. with vsum, vmean, etc.
the v names being understandable by Calc.
INVOLVED is a list to which column numbers of columns
referenced by formula are added."
(replace-regexp-in-string
(rx (quotedcolname nakedname) (? (* space) "("))
(lambda (var)
(save-match-data ;; save because we are called within a replace-regexp
(if (string-match
(rx (group (+ (notany "("))) (* space) "(")
var)
(if (member
(match-string 1 var)
'("mean" "meane" "gmean" "hmean" "median" "sum"
"min" "max" "prod" "pvar" "sdev" "psdev"
"corr" "cov" "pcov" "count" "span" "var"))
;; aggregate functions with or without the leading "v"
;; for example, sum(X) and vsum(X) are equivalent
(format "v%s" var)
var)
;; replace VAR if it is a column name
(let ((i (orgtbl-aggregate--colname-to-int
var
table)))
(if i
(progn
(unless
(memq i (orgtbl-aggregate--list-get involved))
(orgtbl-aggregate--list-append involved i))
(format "Frux(%s)" i))
var)))))
formula
t ;; if nil, Frux is sometimes converted to FRUX
))
(defun orgtbl-aggregate--frux-to-$ (frux)
"Replace all occurences of Frux(NN) by $NN in FRUX"
(replace-regexp-in-string
(rx "Frux(" (group (+ digit)) ")")
(lambda (var)
(format "$%s" (match-string 1 var))
)
frux))
;; dynamic binding
(defvar orgtbl-aggregate--var-keycols)
(cl-defstruct
(orgtbl-aggregate--outcol
;; (:predicate nil) ; worse with this directive
(:copier nil))
;; (formula nil :readonly t) ;; :readonly has no effect
formula ; user-entered formula to compute output cells
format ; user-entered formatter of output cell
sort ; user-entered sorting instruction for output column
invisible ; user-entered output column invisibility
name ; user-entered output column name
formula$ ; derived formula with $N instead of input column names
formula-frux ; derived formula in Calc format with Frux(N) for input columns
involved ; list of input columns numbers appearing in formula
key ; is this output column a key-column?
)
(defun orgtbl-aggregate--parse-col (col table)
"Parse COL specification into an ORGTBL-AGGREGATE--OUTCOL structure.
COL is a column specification. It is a string text:
\"formula;formatter;^sorting;;'alternate_name'\"
If there is no formatter or sorting or other specifier,
nil is given in place. The other fields of orgtbl-aggregate--OUTCOL are
filled here too, and nowhere else.
TABLE is used to convert a column name
into the column number."
;; parse user specification
(unless (stringp col)
(setq col (format "%s" col)))
(unless (string-match
(rx
bos
(group-n 1 (+ (quotedcolname (notany " ;'\""))))
(*
";"
(or
(seq (group-n 2 (* (notany "^;'\"<"))))
(seq "^" (group-n 3 (* (notany "^;'\"<"))))
(seq "<" (group-n 4 (* (notany "^;'\">"))) ">")
(seq "'" (group-n 5 (* (notany "'"))) "'")))
eos)
col)
(user-error "Bad column specification: %S" col))
(let* ((formula (match-string 1 col))
(format (match-string 2 col))
(sort (match-string 3 col))
(invisible (match-string 4 col))
(name (match-string 5 col))
;; list the input column numbers which are involved
;; into formula
(involved (orgtbl-aggregate--list-create))
;; create a derived formula in Calc format,
;; where names of input columns are replaced with
;; frux(N)
(frux (orgtbl-aggregate--to-frux formula table involved))
;; create a derived formula where input column names
;; are replaced with $N
(formula$ (orgtbl-aggregate--frux-to-$ frux))
;; if a formula is just an input column name,
;; then it is a key-grouping-column
(key
(if (string-match-p
(rx bos (quotedcolname nakedname) eos)
formula)
(orgtbl-aggregate--colname-to-int formula table t))))
(if key (push key orgtbl-aggregate--var-keycols))
(make-orgtbl-aggregate--outcol
:formula formula
:format format
:sort sort
:invisible invisible
:name name
:formula$ formula$
:formula-frux (math-read-expr frux)
:involved (orgtbl-aggregate--list-get involved)
:key key)))
;; dynamic binding
(defvar orgtbl-aggregate--columns-sorting)
(cl-defstruct
(orgtbl-aggregate--sorting
;; (:predicate nil) ; worse with this directive
(:copier nil))
;; (strength nil :readonly t) ;; :readonly has no effect
strength ; the 3 in a user specification like ;^a3
colnum ; the number of the output column to sort
ascending ; ;^n is ascending, ;^N is descending
extract ; extract Lisp function, eg string-to-number for ;^n
compare ; comparison Lisp function for 2 cells, eg string< for ;^a
)
(defun orgtbl-aggregate--prepare-sorting (aggcols)
"Create a list of columns to be sorted.
Columns are searched into AGGCOLS.
The resulting list will be used by
`orgtbl-aggregate--columns-sorting'.
The list contains sorting specifications as follows:
. sorting strength
. column number
. ascending descending
. extract function
. compare function
- sorting strength is a number telling what column should be
considered first:
. lower number are considered first
. nil are condirered last
- column number is as in the user specification
1 is the first user specified column
- ascending descending is nil for ascending, t for descending
- extract function converts the input cell (which is a string)
into a comparable value
- compare function compares two cells and answers nil if
the first cell must come before the second."
(cl-loop for col in aggcols
for sorting = (orgtbl-aggregate--outcol-sort col)
for colnum from 0
if sorting
do
(unless (string-match
(rx bol
(group (any "aAnNtTfF"))
(group (* digit))
eol)
sorting)
(user-error
"Bad sorting specification: ^%s, expecting a/A/n/N/t/T and an optional number"
sorting))
(orgtbl-aggregate--list-append
orgtbl-aggregate--columns-sorting
(let ((strength
(if (string= (match-string 2 sorting) "")
nil
(string-to-number (match-string 2 sorting)))))
(pcase (match-string 1 sorting)
("a" (record 'orgtbl-aggregate--sorting strength colnum nil #'orgtbl-aggregate--cell-to-string #'string-lessp))
("A" (record 'orgtbl-aggregate--sorting strength colnum t #'orgtbl-aggregate--cell-to-string #'string-lessp))
("n" (record 'orgtbl-aggregate--sorting strength colnum nil #'orgtbl-aggregate--cell-to-number #'<))
("N" (record 'orgtbl-aggregate--sorting strength colnum t #'orgtbl-aggregate--cell-to-number #'<))
("t" (record 'orgtbl-aggregate--sorting strength colnum nil #'orgtbl-aggregate--cell-to-time #'<))
("T" (record 'orgtbl-aggregate--sorting strength colnum t #'orgtbl-aggregate--cell-to-time #'<))
((or "f" "F") (user-error "f/F sorting specification not (yet) implemented"))
(_ (user-error "Bad sorting specification ^%s" sorting))))))
;; major sorting columns must come before minor sorting columns
(setq orgtbl-aggregate--columns-sorting
(sort (orgtbl-aggregate--list-get orgtbl-aggregate--columns-sorting)
(lambda (a b)
(if (null (orgtbl-aggregate--sorting-strength a))
(and (null (orgtbl-aggregate--sorting-strength b))
(< (orgtbl-aggregate--sorting-colnum a)
(orgtbl-aggregate--sorting-colnum b)))
(or (null (orgtbl-aggregate--sorting-strength b))
(< (orgtbl-aggregate--sorting-strength a)
(orgtbl-aggregate--sorting-strength b))
(and (= (orgtbl-aggregate--sorting-strength a)
(orgtbl-aggregate--sorting-strength b))
(< (orgtbl-aggregate--sorting-colnum a)
(orgtbl-aggregate--sorting-colnum b)))))))))
;; escape lexical binding to eval user given
;; Lisp expression
(defvar orgtbl-aggregate--row)
(defun orgtbl-aggregate--table-add-group (groups hgroups row aggcond)
"Add the source ROW to the GROUPS of rows.
If ROW fits a group within GROUPS, then it is added at the end
of this group.
Otherwise a new group is added at the end of GROUPS,
containing this single ROW.
AGGCOND is a formula which is evaluated against ROW.
If nil, ROW is just discarded.
HGROUPS contains the same information as GROUPS, stored in
a hash-table, whereas GROUPS is a Lisp list."
(and (or (not aggcond)
(let ((orgtbl-aggregate--row row))
;; this eval need the variable 'orgtbl-aggregate--row
;; to have a value
(eval aggcond)))
(let ((gr (gethash row hgroups)))
(unless gr
(setq gr (orgtbl-aggregate--list-create))
(puthash row gr hgroups)
(orgtbl-aggregate--list-append groups gr))
(orgtbl-aggregate--list-append gr row))))
(defun orgtbl-aggregate--read-calc-expr (expr)
"Interpret EXPR (a string) as either an org date or a calc expression."
(cond
;; nil happens when a table is malformed
;; some columns are missing in some rows
((not expr) nil)
;; already an integer? return it
((integerp expr) expr)
;; a floating point? must convert it to Calc
((numberp expr) (math-read-number (number-to-string expr)))
;; empty cell returned as nil,
;; to be processed later depending on modifier flags
((string= expr "") nil)
;; the purely numerical cell case arises very often
;; short-circuiting general functions boosts performance (a lot)
((and
(string-match-p
(rx bos
(? (any "+-")) (* digit)
(? "." (* digit))
(? "e" (? (any "+-")) (+ digit))
eos)
expr)
(not (string-match-p (rx bos (* (any "+-.")) "e") expr)))
(math-read-number expr))
;; Convert an Org-mode date to Calc internal representation
((string-match-p org-ts-regexp0 expr)
(math-parse-date
(replace-regexp-in-string (rx (any "[<>].a-zA-Z")) " " expr)))
;; Convert a duration into a number of seconds
((string-match
(rx bos
(group (+ digit))
":"
(group digit digit)
(? ":" (group digit digit))
eos)
expr)
(+
(* 3600 (string-to-number (match-string 1 expr)))
(* 60 (string-to-number (match-string 2 expr)))
(if (match-string 3 expr) (string-to-number (match-string 3 expr)) 0)))
;; generic case: symbolic calc expression
(t
(math-simplify
(calcFunc-expand
(math-read-expr expr))))))
(defun orgtbl-aggregate--hash-test-equal (row1 row2)
"Are ROW1 & ROW2 equal regarding the key columns?"
(cl-loop for idx in orgtbl-aggregate--var-keycols
always (equal (nth idx row1) (nth idx row2))))
;; Use standard sxhash-equal to hash strings
;; Unfortunately sxhash-equal is weak.
;; So we compensate with multiplications and reminders,
;; while trying to stay within the 2^29 fixnums.
;; see (info "(elisp) Integer Basics")
(defun orgtbl-aggregate--hash-test-hash (row)
"Compute a hash code for ROW from key columns."
(let ((h 456542153))
(cl-loop for idx in orgtbl-aggregate--var-keycols
do
(setq
h
(*
(%
(*
(%
(logxor
(sxhash-equal (nth idx row))
h)
53639)
9973)
53633)
10007)))
h))
(defun orgtbl-aggregate--parse-preprocess (formulas width)
"Parse the :precompute parameter's value, FORMULAS.
WIDTH is the number of columns of the input table, without
the precomputed columns.
Return a list:
((formula1 . name1) (formula2 . name2) …)"
;; formulas is a list of strings
;; change it to ((formula1 . name1) (formula2 . name2) …)
;; name1 etc. default to a dollar name like "$8"
(if (stringp formulas)
(setq formulas
(split-string
formulas
(rx (* space) "::" (* space))
t)))
(cl-loop
for formula in formulas
for i from (1+ width)
for dollari = (format "$%s" i)
collect
(if (string-match
(rx bos
(group-n 1 (+ (notany ";"))) ; formula to compute column
(*
";" (* space) ; maybe something after a semicolon
(or
(seq (group-n 2 (+ (notany "^;'\"<")))) ; a formatter
(seq "'" (group-n 3 (* (notany "'"))) "'") ; column name
(seq ""))) ; nothing after semicolon
(* space)
eos)
formula)
(cons
(if (match-string 2 formula) ; case formula;formatter
(format "%s;%s" (match-string 1 formula) (match-string 2 formula))
(match-string 1 formula)) ; case formula without formatter
(or (match-string 3 formula) dollari)) ; new column's name
(cons formula dollari))))
(defun orgtbl-aggregate--enrich-table (table formulas)
"Enrich TABLE with new columns computed by FORMULAS.
The FORMULAS are supposed to be those used in spreadsheets,
as given after the #+TBLFM: tag.
Actually, FORMULAS are evaluated by Org, not by orgtbl-aggregate."
(let ((start (point))
(width (length (car table))))
(setq
formulas
(orgtbl-aggregate--parse-preprocess formulas width))
(if (memq 'hline table)
;; table has a header? add it the names of the new columns
(cl-loop
for formula in formulas
do (nconc (car table) (list (cdr formula))))
;; table does not have a header? add one of the form:
;; ("$1" "$2" … "$7" "name1" "name2" …)
(setq
table
(cons
(append
(cl-loop for i from 1 to width collect (format "$%s" i))
(cl-loop for formula in formulas collect (cdr formula)))
(cons 'hline table))))
(insert "\n#+TBLFM: ")
(let ((involved (orgtbl-aggregate--list-create)))
(cl-loop
for formula in formulas
for i from (1+ width)
do
(insert
(format
"::$%s=%s"
i
(orgtbl-aggregate--frux-to-$
(orgtbl-aggregate--to-frux (car formula) table involved))))))
(forward-line -1)
(orgtbl-aggregate--insert-elisp-table table)
;; ask Org to evaluate the formulas and fill the new columns
(orgtbl-aggregate--recalculate-fast)
(prog1
;; recover the enriched table a Lisp structure
(orgtbl-aggregate--table-to-lisp)
;; leave the buffer space between #+begin: and #+end:
;; as empty as it was prior to entering this function
(delete-region
start
(let ((case-fold-search t))
(search-forward-regexp (rx bol "#+end:" eol))
(beginning-of-line)
(point)))
(insert "\n")
(goto-char start))))
(defun orgtbl-aggregate--add-hlines (result hline)
"Add hlines to RESULT between different blocks of rows.
HLINE is a small number (1 or 2 or 3, maybe more)
which gives the number of sorted columns to consider
to split rows blocks with hlines.
hlines are added in-place"
(let ((colnums
(cl-loop for col in orgtbl-aggregate--columns-sorting
for n from 1 to hline
collect (orgtbl-aggregate--sorting-colnum col))))
(cl-loop for row on result
unless
(or (null oldrow)
(cl-loop for c in colnums
always
(equal
(nth c (orgtbl-aggregate--list-get (car row)))
(nth c (orgtbl-aggregate--list-get (car oldrow))))))
do (setcdr oldrow (cons 'hline (cdr oldrow)))
for oldrow = row)))
(defun orgtbl-aggregate--create-table-aggregated (table params)
"Convert the source TABLE into an aggregated table.
The source TABLE is a list of lists of cells.
The resulting table follows the specifications,
found in PARAMS entry :cols, ignoring source rows
which do not pass the filter found in PARAMS entry :cond."
(orgtbl-aggregate--pop-leading-hline table)
(define-hash-table-test
'orgtbl-aggregate--hash-test-name
#'orgtbl-aggregate--hash-test-equal
#'orgtbl-aggregate--hash-test-hash)
(let ((groups (orgtbl-aggregate--list-create))
(hgroups (make-hash-table :test 'orgtbl-aggregate--hash-test-name))
(aggcols (plist-get params :cols))
(aggcond (plist-get params :cond))
(hline (plist-get params :hline))
(precompute (plist-get params :precompute))
;; a global variable, passed to the sort predicate
(orgtbl-aggregate--columns-sorting (orgtbl-aggregate--list-create))
;; another global variable
(orgtbl-aggregate--var-keycols))
(if precompute
(setq table (orgtbl-aggregate--enrich-table table precompute)))
(unless aggcols
(setq aggcols (orgtbl-aggregate--get-header-table table)))
(if (stringp aggcols)
(setq aggcols (orgtbl-aggregate--split-string-with-quotes aggcols)))
(cl-loop for col on aggcols
do (setcar col (orgtbl-aggregate--parse-col (car col) table)))
(when aggcond
(if (stringp aggcond)
(setq aggcond (read aggcond)))
(setq aggcond
(orgtbl-aggregate--replace-colnames-nth table aggcond)))
(setq hline
(cond ((null hline)
0)
((numberp hline)
hline)
((string-match-p (rx bos (or "yes" "t") eos) hline)
1)
((string-match-p (rx bos (or "no" "nil") eos) hline)
0)
((string-match-p "[0-9]+" hline)
(string-to-number hline))
(t
(user-error
":hline parameter should be 0, 1, 2, 3, ... or yes, t, no, nil, not %S"
hline))))
;; special case: no sorting column but :hline 1 required
;; then a hidden hline column is added
(if (and (> hline 0)
(cl-loop for col in aggcols
never (orgtbl-aggregate--outcol-sort col)))
(push
(orgtbl-aggregate--parse-col "hline;^n;<>" table)
aggcols))
(orgtbl-aggregate--prepare-sorting aggcols)
; split table into groups of rows
(cl-loop
with nbcols
= (cl-loop
for row in table
maximize (if (listp row) (length row) 0))
with hline = 0
for rownum from 1
for row in
(or (cdr (memq 'hline table)) ;; skip header if any
table)
do
(cond
((eq row 'hline)
(setq hline (1+ hline)))
((listp row)
;; fix too short rows
(if (< (length row) nbcols)
(setq row (nconc row (make-list (- nbcols (length row)) ""))))
(orgtbl-aggregate--table-add-group
groups
hgroups
;; add rownum at beginning and hline at end
(cons rownum (nconc row (list hline)))
aggcond))))
(let ((result ;; pre-allocate all resulting rows
(cl-loop for _x in (orgtbl-aggregate--list-get groups)
collect (orgtbl-aggregate--list-create)))
(all-$list
(cl-loop for _x in (orgtbl-aggregate--list-get groups)
collect
(make-vector
;; + 2 for rownum at 0 & hline at the end
(+ 2 (length (car table)))
nil))))
;; inactivating those two functions boosts performance
(cl-letf (((symbol-function 'math-read-preprocess-string) #'identity)
((symbol-function 'calc-input-angle-units) (lambda (_x) nil)))
;; do aggregation
(cl-loop for coldesc in aggcols
do
(orgtbl-aggregate--compute-sums-on-one-column
groups result coldesc all-$list)))
;; sort table according to columns described in
;; orgtbl-aggregate--columns-sorting
(if orgtbl-aggregate--columns-sorting ;; are there sorting instructions?
(setq result (sort result #'orgtbl-aggregate--sort-predicate)))
;; add hlines if requested
(if (> hline 0)
(orgtbl-aggregate--add-hlines result hline))
(push 'hline result)
;; add other lines of the original header, if any;
;; this is done only if the aggregated column refers to
;; a single source column (either a key column or within
;; an aggregated formula)
(orgtbl-aggregate--pop-leading-hline table)
(if (memq 'hline table)
(cl-loop
for i from (cl-loop
for i from -1
for x in table
until (eq x 'hline)
finally return i)
downto 1
do (push
(cons
nil
(cl-loop for column in aggcols
collect
(if (eq (length (orgtbl-aggregate--outcol-involved column)) 1)
(let ((n (1- (car (orgtbl-aggregate--outcol-involved column)))))
(if (>= n 0)
(nth n (nth i table))
""))
"")))
result)))
;; add the header to the resulting table with column names
;; as they appear in :cols but without decorations
(push
(cons
nil
(cl-loop for column in aggcols
collect (or
(orgtbl-aggregate--outcol-name column)
(replace-regexp-in-string
"['\"]" ""
(orgtbl-aggregate--outcol-formula column)))))
result)
;; remove invisible columns by modifying the table in-place
;; beware! it assumes that the actual list in orgtbl-aggregate--lists
;; is pointed to by the cdr of the orgtbl-aggregate--list
(if (cl-loop for col in aggcols
thereis (orgtbl-aggregate--outcol-invisible col))
(cl-loop for row in result
if (consp row)
do (cl-loop for col in aggcols
with cel = row
if (orgtbl-aggregate--outcol-invisible col)
do (setcdr cel (cddr cel))
else do (orgtbl-aggregate--pop-simple cel))))
;; change appendable-lists to regular lists
(cl-loop for row on result
if (consp (car row))
do (setcar row (orgtbl-aggregate--list-get (car row))))
result)))
(defun orgtbl-aggregate--sort-predicate (rowa rowb)
"Compares ROWA & ROWB (which are Org Mode table rows)
according to orgtbl-aggregate--columns-sorting instructions.
Return nil if ROWA already comes before ROWB."
(setq rowa (orgtbl-aggregate--list-get rowa))
(setq rowb (orgtbl-aggregate--list-get rowb))
(cl-loop for col in orgtbl-aggregate--columns-sorting
for colnum = (orgtbl-aggregate--sorting-colnum col)
for desc = (orgtbl-aggregate--sorting-ascending col)
for extract = (orgtbl-aggregate--sorting-extract col)
for compare = (orgtbl-aggregate--sorting-compare col)
for cella = (funcall extract (nth colnum (if desc rowb rowa)))
for cellb = (funcall extract (nth colnum (if desc rowa rowb)))
thereis (funcall compare cella cellb)
until (funcall compare cellb cella)))
(defun orgtbl-aggregate--cell-to-time (cell)
"Interprete the string CELL into a duration in minutes.
The code was borrowed from org-table.el."
(cond
((numberp cell) cell)
((not (stringp cell))
(error "cell %S is neither a string nor a number to be converted to time"
cell))
((string-match org-ts-regexp-both cell)
(float-time
(org-time-string-to-time (match-string 0 cell))))
((org-duration-p cell) (org-duration-to-minutes cell))
((string-match
(rx bow (+ digit) ":" (= 2 digit) eow)
cell)
(org-duration-to-minutes (match-string 0 cell)))
(t 0)))
(defun orgtbl-aggregate--cell-to-number (cell)
"Convert CELL (a cell in the input table) to a number if it is not already."
(cond
((numberp cell) cell)
((stringp cell) (string-to-number cell))
(t (error "cell %S is not a number neither a string" cell))))
(defun orgtbl-aggregate--fmt-settings (fmt)
"Convert the FMT user-given format.
Result is the FMT-SETTINGS assoc list."
(let ((fmt-settings (plist-put () :fmt nil)))
(when fmt
;; the following code was freely borrowed from org-table-eval-formula
;; not all settings extracted from fmt are used
(while (string-match
(rx (group (any "pnfse")) (group (? "-") (+ digit)))
fmt)
(let ((c (string-to-char (match-string 1 fmt)))
(n (string-to-number (match-string 2 fmt))))
(cl-case c
(?p (setq calc-internal-prec n))
(?n (setq calc-float-format `(float ,n)))
(?f (setq calc-float-format `(fix ,n)))
(?s (setq calc-float-format `(sci ,n)))
(?e (setq calc-float-format `(eng ,n)))))
(setq fmt (replace-match "" t t fmt)))
(while (string-match "[tTUNLEDRFSuQqCc]" fmt)
(cl-case (string-to-char (match-string 0 fmt))
(?t (plist-put fmt-settings :duration t)
(plist-put fmt-settings :numbers t)
(plist-put fmt-settings :duration-output-format org-table-duration-custom-format))
(?T (plist-put fmt-settings :duration t)
(plist-put fmt-settings :numbers t)
(plist-put fmt-settings :duration-output-format nil))
(?U (plist-put fmt-settings :duration t)
(plist-put fmt-settings :numbers t)
(plist-put fmt-settings :duration-output-format 'hh:mm))
(?N (plist-put fmt-settings :numbers t))
(?L (plist-put fmt-settings :literal t))
(?E (plist-put fmt-settings :keep-empty t))
(?D (setq calc-angle-mode 'deg))
(?R (setq calc-angle-mode 'rad))
(?F (setq calc-prefer-frac t))
(?S (setq calc-symbolic-mode t))
(?u (setq calc-simplify-mode 'units))
(?c (plist-put fmt-settings :debug ?c))
(?C (plist-put fmt-settings :debug ?C))
(?q (plist-put fmt-settings :debug ?q))
(?Q (plist-put fmt-settings :debug ?Q)))
(setq fmt (replace-match "" t t fmt)))
(when (string-match-p (rx (not (syntax whitespace))) fmt)
(plist-put fmt-settings :fmt fmt)))
fmt-settings))
(eval-when-compile
(defmacro orgtbl-aggregate--calc-setting (setting &optional setting0)
"Retrieve a Calc setting.
The setting comes either from `org-calc-default-modes'
or from SETTING itself.
SETTING0 is a default to use if both fail."
;; plist-get would be fine, except that there is no way
;; to distinguish a value of nil from no value
;; so we fallback to memq
`(let ((x (memq (quote ,setting) org-calc-default-modes)))
(if x (cadr x)
(or ,setting ,setting0))))
)
(defun orgtbl-aggregate--compute-sums-on-one-column (groups result coldesc all-$list)
"Apply COLDESC over all GROUPS of rows.
COLDESC is a formula given by the user in :cols,
with an optional format.
Common Calc settings and formats are pre-computed before
actually computing sums, because they are the same for all groups.
RESULT is the list of expected resulting rows.
At the beginning, all rows are empty lists.
A cell is appended to every row at each call of this function."
;; within this (let), we locally set Calc settings that must be active
;; for all the calls to Calc:
;; (orgtbl-aggregate--read-calc-expr) and (math-format-value)
(let ((calc-internal-prec
(orgtbl-aggregate--calc-setting calc-internal-prec))
(calc-float-format
(orgtbl-aggregate--calc-setting calc-float-format ))
(calc-angle-mode
(orgtbl-aggregate--calc-setting calc-angle-mode ))
(calc-prefer-frac
(orgtbl-aggregate--calc-setting calc-prefer-frac ))
(calc-symbolic-mode
(orgtbl-aggregate--calc-setting calc-symbolic-mode))
(calc-date-format
(orgtbl-aggregate--calc-setting calc-date-format '(YYYY "-" MM "-" DD " " www (" " hh ":" mm))))
(calc-display-working-message
(orgtbl-aggregate--calc-setting calc-display-working-message))
(fmt-settings nil)
(case-fold-search nil))
;; get that out of the (let) because its purpose is to override
;; what the (let) has set
(setq fmt-settings
(orgtbl-aggregate--fmt-settings
(orgtbl-aggregate--outcol-format coldesc)))
(cl-loop for group in (orgtbl-aggregate--list-get groups)
for row in result
for $list in all-$list
do
(orgtbl-aggregate--list-append
row
(orgtbl-aggregate--compute-one-sum
group
coldesc
fmt-settings
$list)))))
(defun orgtbl-aggregate--compute-one-sum (group coldesc fmt-settings $list)
"Apply a user given formula to one GROUP of input rows.
COLDESC is a structure where several parameters are packed:
see (cl-defstruct orgtbl-aggregate--outcol ...).
Those parameters all describe a single column.
The formula is contained in COLDESC-formula-frux.
Column names have been replaced by Frux(1), Frux(2), Frux(3)... forms.
Those Frux(N) froms are placeholders that will be replaced
by Calc vectors of values extracted from the input table,
in column N.
COLDESC-involved is a list of columns numbers used by COLDESC-formula-frux.
$LIST is a Lisp-vector of Calc-vectors of values from the input table
parsed by Calc. $LIST acts as a cache. When a value is missing, it is
computed, and stored in $LIST. But if there is already a value,
a re-computation is saved.
FMT-SETTINGS are formatter settings computed by
`orgtbl-aggregate--fmt-settings', from user given formatting instructions.
Return an output cell.
When coldesc-key is non-nil, then a key-column is considered,
and a cell from any row in the group is returned."
(cond
;; key column
((orgtbl-aggregate--outcol-key coldesc)
(nth (orgtbl-aggregate--outcol-key coldesc)
(car (orgtbl-aggregate--list-get group))))
;; do not evaluate, output Calc formula
((eq (plist-get fmt-settings :debug) ?c)
(orgtbl-aggregate--outcol-formula$ coldesc))
;; do not evaluate, output Lisp formula
((eq (plist-get fmt-settings :debug) ?q)
(orgtbl-aggregate--outcol-formula-frux coldesc))
;; vlist($3) alone, without parenthesis or other decoration
((string-match
(rx bos (? ?v) "list"
(* blank) "(" (* blank)
"$" (group (+ digit))
(* blank) ")" (* blank) eos)
(orgtbl-aggregate--outcol-formula$ coldesc))
(mapconcat
#'identity ;; there is fast path when `identity' is requested
(cl-loop with i =
(string-to-number
(match-string 1 (orgtbl-aggregate--outcol-formula$ coldesc)))
for row in (orgtbl-aggregate--list-get group)
collect (orgtbl-aggregate--cell-to-string (nth i row)))
", "))
(t
;; all other cases: handle them to Calc
(let ((calc-dollar-values-oo
(orgtbl-aggregate--make-calc-$-list
group
fmt-settings
(orgtbl-aggregate--outcol-involved coldesc)
$list))
(calc-command-flags nil)
(calc-next-why nil)
(calc-language 'flat)
(calc-dollar-used 0))
(let ((ev
(orgtbl-aggregate--defrux
(orgtbl-aggregate--outcol-formula-frux coldesc)
calc-dollar-values-oo
(length (orgtbl-aggregate--list-get group)))))
(cond
((eq (plist-get fmt-settings :debug) ?C)
(math-format-value ev))
((eq (plist-get fmt-settings :debug) ?Q)
(format "%S" ev))
((progn
(setq ev
(math-format-value
(math-simplify
(calcFunc-expand ; yes, double expansion
(calcFunc-expand ; otherwise it is not fully expanded
(math-simplify
ev))))
1000))
(plist-get fmt-settings :fmt))
(format (plist-get fmt-settings :fmt) (string-to-number ev)))
((plist-get fmt-settings :duration)
(org-table-time-seconds-to-string
(string-to-number ev)
(plist-get fmt-settings :duration-output-format)))
(t ev)))))))
(defun orgtbl-aggregate--defrux (formula-frux calc-dollar-values-oo count)
"Replace all Frux(N) expressions in FORMULA-FRUX.
Replace with Calc-vectors found in CALC-DOLLAR-VALUES-OO.
Also replace vcount() forms with the actual number of rows
in the current group, given by COUNT."
(cond
((not (consp formula-frux))
formula-frux)
((eq (car formula-frux) 'calcFunc-Frux)
(nth (cadr formula-frux) calc-dollar-values-oo))
((eq (car formula-frux) 'calcFunc-vcount)
count)
(t
(cl-loop
for x in formula-frux
collect (orgtbl-aggregate--defrux x calc-dollar-values-oo count)))))
(defun orgtbl-aggregate--make-calc-$-list (group fmt-settings involved $list)
"Prepare a list of vectors that will use to replace Frux(N) expressions.
Frux(1) will be replaced by the first element of list,
Frux(2) by the second an so on.
The vectors follow the Calc syntax: (vec a b c ...).
They contain values extracted from rows of the current GROUP.
Vectors are created only for column numbers in INVOLVED.
In FMT-SETTINGS, :keep-empty is a flag to tell whether an empty cell
should be converted to NAN or ignored.
:numbers is a flag to replace non numeric values by 0."
(cl-loop
for i in involved
unless (aref $list i)
do (aset
$list i
(cons 'vec
(cl-loop for row in (orgtbl-aggregate--list-get group)
collect
(orgtbl-aggregate--read-calc-expr (nth i row))))))
(cl-loop
for vec across $list
for i from 0
collect
(when (memq i involved)
(let ((vecc
(if (plist-get fmt-settings :keep-empty)
(cl-loop for x in vec
collect (if x x '(var nan var-nan)))
(cl-loop for x in vec
if x
collect x))))
(if (plist-get fmt-settings :numbers)
(cl-loop for x on (cdr vecc)
unless (math-numberp (car x))
do (setcar x 0)))
vecc))))
;; aggregation in Push mode
;;;###autoload
(defun orgtbl-to-aggregated-table (table params)
"Convert the Org Mode TABLE to an aggregated version.
The resulting table contains aggregated material.
Grouping of rows is done for identical values of grouping columns.
For each group, aggregation (sum, mean, etc.) is done for other columns.
The source table must contain sending directives with the following format:
#+ORGTBL: SEND destination orgtbl-to-aggregated-table :cols ... :cond ...
The destination must be specified somewhere in the same file
with a block like this:
#+BEGIN RECEIVE ORGTBL destination
#+END RECEIVE ORGTBL destination
PARAMS are parameters given in the #+ORGTBL: SEND line.
:cols gives the specifications of the resulting columns.
It is a space-separated list of column specifications.
Example:
P Q sum(X) max(X) mean(Y)
Which means:
group rows with similar values in columns P and Q,
and for each group, compute the sum of elements in
column X, etc.
The specification for a resulting column may be:
COL the name of a grouping column in the source table
hline a special name for grouping rows separated
by horizontal lines
count() give the number of rows in each group
list(COL) list the values of the column for each group
sum(COL) sum of the column for each group
sum(COL1*COL2) sum of the product of two columns for each group
mean(COL) average of the column for each group
mean(COL1*COL2) average of the product of two columns per group
meane(COL) average and estimated error
hmean(COL) harmonic average
gmean(COL) geometric average
median(COL) middle element after sorting them in each group
max(COL) largest element of each group
min(COL) smallest element of each group
sdev(COL) standard deviation (divide by N-1)
psdev(COL) population standard deviation (divide by N)
pvar(COL) variance of values in each group
prod(COL) product of values in each group
cov(COL1,COL2) covariance of two columns, per group (div. by N-1)
pcov(COL1,COL2) population covariance of two columns (div. by N)
corr(COL1,COL2) linear correlation of two columns
:cond optional
A lisp expression to filter out rows in the source table.
When the expression evaluate to nil for a given row of the
source table, then this row is discarded in the resulting table.
Example:
(equal Q \"b\")
Which means: keep only source rows for which the column Q
has the value b
Names of columns in the source table may be in the dollar form,
for example use $3 to name the 3th column,
or by its name if the source table have a header.
If all column names are in the dollar form,
the table is supposed not to have a header.
The special column name \"hline\" takes values from zero and up
and is incremented by one for each horizontal line.
Example:
add a line like this one before your table
,#+ORGTBL: SEND aggregatedtable orgtbl-to-aggregated-table \\
:cols \"sum(X) q sum(Y) mean(Z) sum(X*X)\"
then add somewhere in the same file the following lines:
,#+BEGIN RECEIVE ORGTBL aggregatedtable
,#+END RECEIVE ORGTBL aggregatedtable
Type \\ & \\[org-ctrl-c-ctrl-c] into your source table
Note:
This is the \"push\" mode for aggregating a table.
To use the \"pull\" mode, look at the org-dblock-write:aggregate function.
Note:
The name `orgtbl-to-aggregated-table' follows the Org Mode standard
with functions like `orgtbl-to-csv', `orgtbl-to-html'..."
(interactive)
(orgtbl-aggregate--elisp-table-to-string
(orgtbl-aggregate--post-process
(orgtbl-aggregate--create-table-aggregated table params)
(plist-get params :post))))
;; aggregation in Pull mode
(defun orgtbl-aggregate--remove-cookie-lines (table)
"Remove lines of TABLE which contain cookies.
But do not remove cookies in the header, if any.
The operation is destructive. But on the other hand,
if there are no cookies in TABLE, TABLE is returned
without any change.
A cookie is an alignment instruction like:
left align cells in this column
center cells
right align
<15> make this column 15 characters wide."
(orgtbl-aggregate--pop-leading-hline table)
(cl-loop
with hline = nil
for line on table
if (and hline
(cl-loop
for cell in (car line)
thereis
(and (stringp cell)
(string-match-p
(rx bos "<" (? (any "lcr")) (* digit) ">" eos)
cell))))
do (setcar line t)
if (eq (car line) 'hline)
do (setq hline t))
(delq t table))
;;;###autoload
(defun org-dblock-write:aggregate (params)
"Create a table which is the aggregation of material from another table.
Grouping of rows is done for identical values of grouping columns.
For each group, aggregation (sum, mean, etc.) is done for other columns.
PARAMS contains user parameters given on the #+BEGIN: aggregate line,
as follow:
:table name of the source table
:cols gives the specifications of the resulting columns.
It is a space-separated list of column specifications.
Example:
\"P Q sum(X) max(X) mean(Y)\"
Which means:
group rows with similar values in columns P and Q,
and for each group, compute the sum of elements in
column X, etc.
The specification for a resulting column may be:
COL the name of a grouping column in the source table
hline a special name for grouping rows separated
by horizontal lines
count() number of rows in each group
list(COL) list the values of the column for each group
sum(COL) sum of the column for each group
sum(COL1*COL2) sum of the product of two columns for each group
mean(COL) average of the column for each group
mean(COL1*COL2) average of the product of two columns per group
meane(COL) average along with the estimated error per group
hmean(COL) harmonic average per group
gmean(COL) geometric average per group
median(COL) middle element after sorting them, per group
max(COL) largest element of each group
min(COL) smallest element of each group
sdev(COL) standard deviation (divide by N-1)
psdev(COL) population standard deviation (divide by N)
pvar(COL) variance per group
prod(COL) product per group
cov(COL1,COL2) covariance of two columns per group (div. by N-1)
pcov(COL1,COL2) population covariance of two columns (div. by N)
corr(COL1,COL2) linear correlation of two columns, per group
:cond optional
A Lisp expression to filter out rows in the source table.
When the expression evaluate to nil for a given row of
the source table, then this row is discarded in the resulting table
Example:
(equal Q \"b\")
Which means: keep only source rows for which the column Q
has the value b.
Names of columns in the source table may be in the dollar form,
for example $3 to name the 3th column,
or by its name if the source table have a header.
If all column names are in the dollar form,
the table is supposed not to have a header.
The special column name \"hline\" takes values from zero and up
and is incremented by one for each horizontal line.
Example:
- Create an empty dynamic block like this:
#+BEGIN: aggregate :table originaltable \\
:cols \"sum(X) Q sum(Y) mean(Z) sum(X*X)\"
#+END
- Type \\ & \\[org-ctrl-c-ctrl-c] over the BEGIN line
this fills in the block with an aggregated table
Note:
This is the \"pull\" mode for aggregating a table.
To use the \"push\" mode,
look at the `orgtbl-to-aggregated-table' function.
Note:
The name `org-dblock-write:aggregate' is constrained
by the `org-update-dblock' function."
(interactive)
(let ((formula (plist-get params :formula))
(content (plist-get params :content))
(post (plist-get params :post)))
(if content
(let ((case-fold-search t))
(string-match
(rx bos
(* (* blank) "\n")
(group (* (* blank) (? "#+" (* nonl)) "\n")))
content)
(insert
(replace-regexp-in-string
(rx bol (* (or blank "\n")) eos)
""
(replace-regexp-in-string
(rx bol "#+tblfm" (* (or any "\n")) eos)
""
(match-string 1 content))))))
(orgtbl-aggregate--insert-elisp-table
(orgtbl-aggregate--post-process
(orgtbl-aggregate--create-table-aggregated
(orgtbl-aggregate--remove-cookie-lines
(orgtbl-aggregate-table-from-any-ref (plist-get params :table)))
params)
post))
(orgtbl-aggregate--table-recalculate content formula)))
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Wizard
;; This variable contains the history of user entered answers,
;; so that they can be entered again or edited.
(defvar orgtbl-aggregate-history-cols ())
(defun orgtbl-aggregate--parse-header-arguments (type)
"If (point) is on a #+begin: line, parse it, and return a plist.
TYPE is \"aggregate\", or possibly any type of block.
If the line the (point) is on do not match TYPE, return nil."
(let ((line (buffer-substring-no-properties
(line-beginning-position)
(line-end-position)))
(case-fold-search t))
(and
(string-match
(rx bos (* blank) "#+begin:" (* blank) (group (+ word)) (group (* nonl)) eos)
line)
(equal (match-string 1 line) type)
(let ((list (read (concat "(" (match-string 2 line) ")"))))
(cl-loop
for pair on list
for val = (cadr pair)
do
(if (symbolp val)
(setcar (cdr pair) (symbol-name val)))
(setq pair (cdr pair)))
list))))
(defun orgtbl-aggregate--dismiss-help ()
"Hide the wizard help window."
(let ((help-window (get-buffer-window "*orgtbl-aggregate-help*")))
(if help-window
(delete-window help-window))))
(defun orgtbl-aggregate--display-help (explain &rest args)
"Display help for each field the wizard queries.
EXPLAIN is a text in Org Mode to display. It is process
through `format' with replacements in ARGS."
(let ((docs
'(
:isorgid "* Input table locator
The input table may be pointed to by:
- a file and a name,
- an Org Mode identifier"
:orgid "* Org ID
It is an identifier hidden in a =properties= drawer.
Org Mode globally keeps track of all Ids and knows how to access them.
It is supposed that the ID location is followed by a table or
a Babel block suitable for aggregation."
:name "* The input table may be:
- a regular Org table,
- a Babel block whose output will be the input table.
Org table & Babel block names are available at completion (type ~TAB~).
Leave empty for a CSV or JSON formatted table."
:params "* Parameters for Babel code block (optional)
** A Babel code block may require specific parameters
Give them here if needed, surrounded by parenthesis. Example:
~(size=4,reverse=nil)~
** CSV or JSON formatted tables.
Examples:
~(csv header)~ ~(json)~
** Regular Org Mode table
Leave empty."
:slice "* Slicing (optional)
Slicing is an Org Mode feature allowing to cut the input table.
It applies to any input: Org table, Babel output, CSV, JSON.
Leave empty for no slicing.
** Examples:
- ~mytable[0:5]~ retains only the first 6 rows of the input table
- ~mytable[*,0:1]~ retains only the first 2 columns
- ~mytable[0:5,0:1]~ retains 5 rows and 2 columns"
:cond "* Filter rows (optional)
Lisp function, lambda, or Babel block to filter out rows.
** Available input columns
%s
** Example
~(>= (string-to-number quty) 3)~
only rows with cell ~quty~ higher or equal to ~3~ are retained.
~(not (equal tag \"dispose\"))~
rows with cell ~tag~ equal to ~dispose~ are filtered out."
:post "* Post-process (optional)
The output table may be post-processed prior to printing it
in the current buffer.
The processor may be a Lisp function, a lambda, or a Babel block.
** Example:
~(lambda (table) (append table '(hline (banana 42))))~
two rows are appended at the end of the output table:
~hline~ which means horizontal line,
and a row with two cells."
;; bazilo]
:file "* In which file is the table?
The table may be in another file.
Leave answer empty to mean that the table is in the current buffer."
:precompute "* Precompute (optional)
The input table may be enriched with additional columns prior to aggregating.
The syntax is the regular Org table spreadsheet formulas for columns,
including formatting.
Additionnaly, the name of new columns can be specified after a semicolumn.
** Available columns
%s
** Example
~quty*10;f1;'q10'~
means:
- add a new column to the input table named ~q10~
- compute it as ~10~ times the ~quty~ input column
- format it with ~f1~, 1 digit after dot"
:cols "* Target columns
** They may be
- bare input columns, acting as grouping keys,
- formulas in the syntax of Org spreadsheet, like ~vmean()~, ~vsum()~, ~count()~.
** Formatting
Each target column may be followed optionally by semicolon separated parameters:
- alternate name, example ;'alternate-name'
- formatting, examples ~;f2~ ~;%%.2f~
- sorting, examples ~;^a~ ~;^A~ ~;^n~ ~;^N~
- invisibility ~;<>~
** Available input columns
%s
** Examples:
~vmean(quty);f2~, ~vsum(amount);'total'~"
:hline "* Output horizontal separators level (optional)
- ~0~ or empty means no lines in the output
- ~1~ means separate rows of identical values on 1 column
- ~2~ means separate rows of identical values on 2 columns
- larger values are allowed, but questionably useful.
The columns considered are the sorted ones."
:cols-tr "* Target columns
** Optional
If the answer is left empty, all input columns are kept,
in the same order.
** Available input columns
%s"
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
))
(main-window (selected-window))
(help-window (get-buffer-window "*orgtbl-aggregate-help*")))
(if help-window
(select-window help-window)
(setq main-window (split-window nil 16 'above))
(switch-to-buffer "*orgtbl-aggregate-help*")
(setq help-window (selected-window)))
(org-mode)
(erase-buffer)
(insert (apply #'format (plist-get docs explain) args))
(goto-char (point-min))
(select-window main-window)))
(defun orgtbl-aggregate--wizard-query-table (table expert)
"Query the 4 fields composing a generalized table: file:name:params:slice.
It may be only 3 fields in case of orgid:params:slice or
file.csv:(csv):slice.
If TABLE is not nil, it is decomposed into file:name:params:slice, and each
of those 4 fields serve as default answer when prompting.
Alternately, file:name may be orgid, an ID which knows its file location.
When EXPERT is nil, only basic parameters are queried.
Note that when an expert parameter was set prior to entering the wizard,
it is queried even when EXPERT is nil."
(let (file name orgid params slice isorgid)
(if table
(let ((struct (orgtbl-aggregate--parse-locator table)))
(setq file (aref struct 0))
(setq name (aref struct 1))
(setq orgid (aref struct 2))
(setq params (aref struct 3))
(setq slice (aref struct 4))))
(setq
isorgid
(cond
(orgid t)
(name nil)
(expert
(orgtbl-aggregate--display-help :isorgid)
(let ((use-short-answers t))
(yes-or-no-p "Is the input pointed to by an Org Mode ID? ")))
(t nil)))
(if isorgid
(progn
(orgtbl-aggregate--display-help :orgid)
(unless org-id-locations (org-id-locations-load))
(setq orgid
(completing-read
"Org ID: "
(hash-table-keys org-id-locations)
nil
nil ;; user is free to input anything
orgid)))
(when (or expert file)
(orgtbl-aggregate--display-help :file)
(let ((insert-default-directory nil))
(setq file
(orgtbl-aggregate--nil-if-empty
(read-file-name "File (RET for current buffer): "
nil
nil
nil
file)))))
(orgtbl-aggregate--display-help :name)
(setq name
(completing-read
"Table or Babel: "
(orgtbl-aggregate--list-local-tables file)
nil
nil ;; user is free to input anything
name)))
(and
file
(not params)
(cond
((string-match-p (rx ".csv" eos) file)
(setq params "(csv)"))
((string-match-p (rx ".json" eos) file)
(setq params "(json)"))))
(when (or expert params)
(orgtbl-aggregate--display-help :params)
(setq params
(read-string
"Babel parameters (optional): "
params
'orgtbl-aggregate-history-cols)))
(when (or expert slice)
(orgtbl-aggregate--display-help :slice)
(setq slice
(read-string
"Input slicing (optional): "
slice
'orgtbl-aggregate-history-cols)))
(orgtbl-aggregate--assemble-locator file name orgid params slice)))
;; bazilo]
(defun orgtbl-aggregate--wizard-aggregate-create-update (oldline expert)
"Update OLDLINE parameters by interactivly querying user.
OLDLINE is a plist containing parameter-value pairs.
Example: \\'(:table \"thetable\" :cols \"day vsum(quty)\" …)
OLDLINE is supposed to be extracted from an Org Mode block such as:
#+begin: aggregate :table \"thetable\" :cols \"day vsum(quty)\" …
If (point) is not on such a line, OLDLINE is nil.
The function returns a plist which is an updated version of OLDLINE
amended by the user.
When EXPERT is nil, only basic parameters are queried.
Note that when an expert parameter was set prior to entering the wizard,
it is queried even when EXPERT is nil."
(let ((minibuffer-local-completion-map
(define-keymap :parent minibuffer-local-completion-map
"SPC" nil)) ;; allow inserting spaces
table headerlist header precompute
aggcols aggcond hline postprocess params)
(save-window-excursion
(setq table
(orgtbl-aggregate--wizard-query-table
(orgtbl-aggregate--plist-get-remove oldline :table)
expert))
(setq headerlist
(orgtbl-aggregate--get-header-table table))
(setq header
(mapconcat
(lambda (x) (format " ~%s~" x))
headerlist))
(setq precompute (orgtbl-aggregate--plist-get-remove oldline :precompute))
(when (or expert precompute)
(orgtbl-aggregate--display-help :precompute header)
(setq precompute
(read-string
"Formulas for additional input columns (optional): "
precompute
'orgtbl-aggregate-history-cols)))
(when (orgtbl-aggregate--nil-if-empty precompute)
(setq headerlist
(append headerlist
(cl-loop
for pair in
(orgtbl-aggregate--parse-preprocess
precompute
(length headerlist))
collect (cdr pair))))
(setq header
(mapconcat
(lambda (x) (format " ~%s~" x))
headerlist)))
(orgtbl-aggregate--display-help :cols header)
(setq aggcols
(replace-regexp-in-string
"\"" "'"
(read-string
"Target columns & formulas: "
(orgtbl-aggregate--merge-list-into-single-string
(orgtbl-aggregate--plist-get-remove oldline :cols))
'orgtbl-aggregate-history-cols)))
(setq aggcond (orgtbl-aggregate--plist-get-remove oldline :cond))
(when (or expert aggcond)
(orgtbl-aggregate--display-help :cond header)
(setq aggcond
(read-string
"Row filter (optional): "
(and aggcond (format "%s" aggcond))
'orgtbl-aggregate-history-cols)))
(setq hline (orgtbl-aggregate--plist-get-remove oldline :hline))
(when (or expert hline)
(orgtbl-aggregate--display-help :hline)
(setq hline
(completing-read
"hline (optional): "
'("0" "1" "2" "3")
nil
'confirm
(orgtbl-aggregate--cell-to-string hline))))
(setq postprocess (orgtbl-aggregate--plist-get-remove oldline :post))
(when (or expert postprocess)
(orgtbl-aggregate--display-help :post)
(setq postprocess
(read-string
"Post process (optional): "
postprocess
'orgtbl-aggregate-history-cols)))
)
(setq params
(list
:name "aggregate"
:table table
:cols aggcols))
(if (orgtbl-aggregate--nil-if-empty aggcond)
(nconc params `(:cond ,(read aggcond))))
(if (orgtbl-aggregate--nil-if-empty hline)
(nconc params `(:hline ,hline)))
(if (orgtbl-aggregate--nil-if-empty precompute)
(nconc params `(:precompute ,precompute)))
(if (orgtbl-aggregate--nil-if-empty postprocess)
(nconc params `(:post ,postprocess)))
;; recover parameters not taken into account by the wizard
(cl-loop
for pair on oldline
if (car pair)
do (nconc params `(,(car pair) ,(cadr pair)))
do (setq pair (cdr pair)))
params))
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
;;;###autoload
(defun orgtbl-aggregate-insert-dblock-aggregate (&optional expert)
"Wizard to interactively insert a dynamic aggregated block.
When EXPERT is nil, only basic parameters are queried.
Note that when an expert parameter was set prior to entering the wizard,
it is queried even when EXPERT is nil."
(interactive "P")
(let* ((oldline (orgtbl-aggregate--parse-header-arguments "aggregate"))
(params
(save-excursion (orgtbl-aggregate--wizard-aggregate-create-update oldline expert))))
(if (not oldline)
(org-create-dblock params)
(beginning-of-line)
(kill-line)
(org-create-dblock params)
(delete-blank-lines)
(forward-line 1)
(kill-line 1)
(forward-line -1)
(delete-blank-lines))
(org-update-dblock)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Unfold, Fold
;; Experimental
;; Typing TAB on a line like
;; #+begin aggregate params…
;; unfolds the parameters: a new line for each parameter
;; and a dedicated help & completion for each activated by TAB
;;;###autoload
(defun orgtbl-aggregate-dispatch-TAB ()
"Type TAB on a line like #+begin: aggregate to activate custom functions.
Actually, any line following this pattern will do:
#+xxxxx: yyyyy
Typing TAB will dispatch to function org-TAB-xxxxx-yyyyy if it exists.
If it does not exist, Org Mode will proceed as usual.
If it exists and returns nil, Org Mode will proceed as usual as well.
It it returns non-nil, the TAB processing will stop there."
(save-excursion
(if (and
(not (bolp))
(progn
(end-of-line)
(not (org-fold-core-folded-p)))
(progn
(beginning-of-line)
(re-search-forward
(rx
point
"#+"
(group (+ (any "a-z0-9_-")))
":"
(* blank)
(group (+ (any ":a-z0-9_-")))
(* blank))
nil t)))
(let ((symb
(intern
(format
"org-TAB-%s-%s"
(downcase (match-string-no-properties 1))
(downcase (match-string-no-properties 2))))))
(if (symbol-function symb)
(funcall symb))))))
(defun org-TAB-begin-aggregate ()
"Dispatch to unfolding or folding code.
If the line following
#+begin: aggregate
is an unfoled one of the form:
#+aggregate: …
then proceed to folding, otherwise unfold."
(if (save-excursion
(forward-line 1)
(beginning-of-line)
(let ((case-fold-search t))
(re-search-forward
(rx point "#+aggregate:")
nil t)))
(org-TAB-begin-aggregate-fold)
(org-TAB-begin-aggregate-unfold)))
(defun orgtbl-aggregate--insert-remove-pair-from-alist (tag alist)
"Helper function for folding a pair (TAG . VALUE) in ALIST."
(let ((value
(orgtbl-aggregate--nil-if-empty
(orgtbl-aggregate--alist-get-remove tag alist))))
(if value
(insert
(format " %s %s"
tag
(prin1-to-string value))))))
(defun orgtbl-aggregate-get-all-unfolded ()
"Prepare an a-list of all unfolded parameters."
(interactive)
(save-excursion
(re-search-backward (rx bol "#+begin:") nil t)
(cl-loop
do (forward-line 1)
while
(let ((case-fold-search t))
(re-search-forward
(rx point "#+aggregate:" (* blank)
(group (+ (any ":a-z0-9_-")))
(* blank)
(group (* nonl)))
nil t))
collect
(cons
(intern (match-string-no-properties 1))
(match-string-no-properties 2)))))
(defun orgtbl-aggregate--TAB-replace-value (getter)
"Update a #+aggregate: line
from
#+aggregate: :tag OLD
to
#+aggregate: :tag NEW
NEW being the result of executing (GETTER OLD)"
(let* ((start (point))
(end (pos-eol))
(new
(funcall
getter
(buffer-substring-no-properties start end))))
(when new
(delete-region start end)
(delete-horizontal-space)
(insert " " new))))
;; bazilo]
(defun org-TAB-begin-aggregate-fold ()
"Turn all lines of the form #+aggregate: … into a single line.
That is, fold the may lines of the form:
#+aggregate: param…
into the single line of the form:
#+begin: aggregate params…
Note that the resulting :table XXX parameter is composed of several
individual parameters."
(orgtbl-aggregate--dismiss-help)
(let* ((alist (orgtbl-aggregate-get-all-unfolded)))
(end-of-line)
(insert
" :table \""
(orgtbl-aggregate--assemble-locator
(orgtbl-aggregate--alist-get-remove :file alist)
(orgtbl-aggregate--alist-get-remove :name alist)
(orgtbl-aggregate--alist-get-remove :orgid alist)
(orgtbl-aggregate--alist-get-remove :params alist)
(orgtbl-aggregate--alist-get-remove :slice alist))
"\"")
(orgtbl-aggregate--insert-remove-pair-from-alist :precompute alist)
(orgtbl-aggregate--insert-remove-pair-from-alist :cols alist)
(orgtbl-aggregate--insert-remove-pair-from-alist :cond alist)
(orgtbl-aggregate--insert-remove-pair-from-alist :hline alist)
(orgtbl-aggregate--insert-remove-pair-from-alist :post alist)
(cl-loop
for pair in alist
if (car pair)
do (orgtbl-aggregate--insert-remove-pair-from-alist (car pair) alist))
(forward-line 1)
(while
(let ((case-fold-search t))
(beginning-of-line)
(re-search-forward (rx point "#+aggregate:") nil t))
(beginning-of-line)
(delete-line))
(forward-line -1)
t))
(defun org-TAB-begin-aggregate-unfold ()
"Turn the single line #+begin: aggregate into several lines.
That is, move all parameters in the line
#+begin: aggregate params…
into several lines, each with a single parameter.
Note that the :table XXX parameter is decomposed into several
individual parameter for an easier reading."
(let* ((line (orgtbl-aggregate--parse-header-arguments "aggregate"))
(point (progn (end-of-line) (point)))
(struct (orgtbl-aggregate--parse-locator
(orgtbl-aggregate--plist-get-remove line :table))))
(insert "\n#+aggregate: :file " (or (aref struct 0) ""))
(insert "\n#+aggregate: :name " (or (aref struct 1) ""))
(insert "\n#+aggregate: :orgid " (or (aref struct 2) ""))
(insert "\n#+aggregate: :params " (or (aref struct 3) ""))
(insert "\n#+aggregate: :slice " (or (aref struct 4) ""))
(insert "\n#+aggregate: :precompute "
(or (orgtbl-aggregate--plist-get-remove line :precompute) ""))
(insert "\n#+aggregate: :cols "
(orgtbl-aggregate--merge-list-into-single-string
(or (orgtbl-aggregate--plist-get-remove line :cols ) "")))
(insert "\n#+aggregate: :cond "
(format "%s" (or (orgtbl-aggregate--plist-get-remove line :cond ) "")))
(insert "\n#+aggregate: :hline "
(format "%s" (or (orgtbl-aggregate--plist-get-remove line :hline) "")))
(insert "\n#+aggregate: :post "
(format "%s" (or (orgtbl-aggregate--plist-get-remove line :post ) "")))
(cl-loop
for pair on line
if (car pair)
do (insert (format "\n#+aggregate: %s %s" (car pair) (cadr pair)))
do (setq pair (cdr pair)))
(goto-char point)
(beginning-of-line)
(forward-word 2)
(delete-region (point) point)
t))
(defun orgtbl-aggregate--column-names-from-unfolded ()
"Return a textual list of column names.
They are computed by looking at the distant table
(an Org table, a Babel block, a CSV, or a JSON)
and recovering its header if any.
If there is no header, $1 $2 $3... is returned."
(let*
((alist (orgtbl-aggregate-get-all-unfolded))
(table
(orgtbl-aggregate--assemble-locator
(alist-get :file alist)
(alist-get :name alist)
(alist-get :orgid alist)
(alist-get :params alist)
(alist-get :slice alist))))
(mapconcat
(lambda (x) (format " ~%s~" x))
(orgtbl-aggregate--get-header-table table))))
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
(defun org-TAB-aggregate-:file ()
"Provide help and completion for the #+aggregate: file XXX parameter."
(orgtbl-aggregate--display-help :file)
(orgtbl-aggregate--TAB-replace-value
(lambda (old)
(read-file-name
"File: "
(file-name-directory old)
nil
nil
(file-name-nondirectory old)))))
(defun org-TAB-aggregate-:name ()
"Provide help and completion for the #+aggregate: name XXX parameter."
(orgtbl-aggregate--display-help :name)
(orgtbl-aggregate--TAB-replace-value
(lambda (old)
(completing-read
"Table or Babel name: "
(orgtbl-aggregate--list-local-tables
(orgtbl-aggregate--nil-if-empty
(alist-get :file (orgtbl-aggregate-get-all-unfolded))))
nil
nil ;; user is free to input anything
old))))
(defun org-TAB-aggregate-:orgid ()
"Provide help and completion for the #+aggregate: id XXX parameter."
(orgtbl-aggregate--display-help :orgid)
(unless org-id-locations (org-id-locations-load))
(orgtbl-aggregate--TAB-replace-value
(lambda (old)
(completing-read
"Org-ID: "
(hash-table-keys org-id-locations)
nil
nil ;; user is free to input anything
old))))
(defun org-TAB-aggregate-:params ()
(orgtbl-aggregate--display-help :params))
(defun org-TAB-aggregate-:slice ()
(orgtbl-aggregate--display-help :slice))
(defun org-TAB-aggregate-:cols ()
(orgtbl-aggregate--display-help :cols
(orgtbl-aggregate--column-names-from-unfolded)))
(defun org-TAB-aggregate-:cond ()
(orgtbl-aggregate--display-help :cond
(orgtbl-aggregate--column-names-from-unfolded)))
(defun org-TAB-aggregate-:post ()
(orgtbl-aggregate--display-help :post))
;; bazilo]
(defun org-TAB-aggregate-:precompute ()
(orgtbl-aggregate--display-help :precompute
(orgtbl-aggregate--column-names-from-unfolded)))
(defun org-TAB-aggregate-:hline ()
"Hitting TAB on #+aggregate: hline N cycles the parameter value.
The cycle is
nothing → 1 → 2 → 3 → nothing."
(orgtbl-aggregate--display-help :hline)
(orgtbl-aggregate--TAB-replace-value
(lambda (old)
(cond
((equal old "" ) "1")
((equal old "1") "2")
((equal old "2") "3")
((equal old "3") "" )
(t "")))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The Transposition package
(defun orgtbl-aggregate--create-table-transposed (table cols aggcond)
"Convert the source TABLE to a tranposed version.
TABLE is a list of lists of cells.
COLS gives the source columns that should become rows.
If COLS is nil, all source columns are taken.
AGGCOND is a Lisp expression given bu the user. It is evaluated
against each row. If the result is nil, the row is ignored.
If AGGCOND is nil, all source rows are taken."
(if (stringp cols)
(setq cols (orgtbl-aggregate--split-string-with-quotes cols)))
(setq cols
(if cols
(cl-loop for column in cols
collect
(orgtbl-aggregate--colname-to-int column table t))
(let ((head table))
(orgtbl-aggregate--pop-leading-hline head)
(cl-loop for _x in (car head)
for i from 1
collect i))))
(if aggcond
(setq aggcond
(orgtbl-aggregate--replace-colnames-nth table aggcond)))
(let ((result (cl-loop for _x in cols collect (list t)))
;; '(t) or `(t) would be incorrect╶────────▷─╯
(nhline 0))
(cl-loop for row in table
for rownum from 0
do
(if (eq row 'hline)
(setq nhline (1+ nhline))
(setq row (cons nhline row)))
do
(when (or (eq row 'hline) (not aggcond) (eval aggcond))
(cl-loop
for spec in cols
for r in result
do
(nconc
r
(list
(cond
((eq row 'hline) "")
((eq spec 0) rownum)
((>= spec (length row)) (nth 0 row))
(t (nth spec row))))))))
(cl-loop for row in result
do (orgtbl-aggregate--pop-simple row)
collect
(if (cl-loop for x in row
always (equal "" x))
'hline
row))))
;;;###autoload
(defun orgtbl-to-transposed-table (table params)
"Convert the Org Mode TABLE to a transposed version.
Rows become columns, columns become rows.
The source table must contain sending directives with
the following format:
#+ORGTBL: SEND destination orgtbl-to-transposed-table :cols ... :cond ...
PARAMS are the user given parameters found in the #+ORGTBL: SEND line
The destination must be specified somewhere in the same file
with a bloc like this:
#+BEGIN RECEIVE ORGTBL destination
#+END RECEIVE ORGTBL destination
:cols optional, if omitted all source columns are taken.
Columns specified here will become rows in the result.
Valid specifications are
- names as they appear in the first row of the source table
- $N forms, starting from $1
- the special hline column which is the numbering of
blocks separated by horizontal lines in the source table
:cond optional
a lisp expression to filter out rows in the source table
when the expression evaluate to nil for a given row of
the source table, then this row is discarded in the
resulting table.
Example:
(equal Q \"b\")
Which means: keep only source rows for which the column Q
has the value b
Columns in the source table may be in the dollar form,
for example $3 to name the 3th column,
or by its name if the source table have a header.
If all column names are in the dollar form,
the table is supposed not to have a header.
The special column name \"hline\" takes values from zero and up
and is incremented by one for each horizontal line.
Horizontal lines are converted to empty columns,
and the other way around.
The destination must be specified somewhere in the same file
with a block like this:
#+BEGIN RECEIVE ORGTBL destination_table_name
#+END RECEIVE ORGTBL destination_table_name
Type \\ & \\[org-ctrl-c-ctrl-c] in the source
table to re-create the transposed version.
Note:
This is the \"push\" mode for transposing a table.
To use the \"pull\" mode, look at the org-dblock-write:transpose function.
Note:
The name `orgtbl-to-transposed-table' follows the Org Mode standard
with functions like `orgtbl-to-csv', `orgtbl-to-html'..."
(interactive)
(orgtbl-aggregate--elisp-table-to-string
(orgtbl-aggregate--post-process
(orgtbl-aggregate--create-table-transposed
table
(plist-get params :cols)
(plist-get params :cond))
(plist-get params :post))))
;;;###autoload
(defun org-dblock-write:transpose (params)
"Create a transposed version of an Org Mode table.
Rows become columns, columns become rows.
PARAMS are the user given parameters found in the
#+BEGIN: transpose line
:table names the source table
:cols optional, if omitted all source columns are taken.
Columns specified here will become rows in the result.
Valid specifications are
- names as they appear in the first row of the source table
- $N forms, starting from $1
- the special hline column which is the numbering of
blocks separated by horizontal lines in the source table.
:cond optional
a Lisp expression to filter out rows in the source table
when the expression evaluate to nil for a given row of the
source table, then this row is discarded in the resulting table.
Example:
(equal q \"b\")
Which means: keep only source rows for which the column q
has the value b.
Columns in the source table may be in the dollar form,
for example $3 to name the 3th column,
or by its name if the source table have a header.
If all column names are in the dollar form,
the table is supposed not to have a header.
The special column name \"hline\" takes values from zero and up
and is incremented by one for each horizontal line.
Horizontal lines are converted to empty columns,
and the other way around.
- Create an empty dynamic block like this:
#+BEGIN: transpose :table originaltable
#+END
- Type \\ & \\[org-ctrl-c-ctrl-c] over the BEGIN line
this fills in the block with the transposed table
Note:
This is the \"pull\" mode for transposing a table.
To use the \"push\" mode, look at the orgtbl-to-transposed-table function.
Note:
The name `org-dblock-write:transpose' is constrained
by the `org-update-dblock' function."
(interactive)
(let ((formula (plist-get params :formula))
(content (plist-get params :content))
(post (plist-get params :post)))
(if content
(let ((case-fold-search t))
(string-match
(rx bos
(* (* blank) "\n")
(group (* (* blank) (? "#+" (* nonl)) "\n")))
content)
(insert
(replace-regexp-in-string
(rx bol (* (or blank "\n")) eos)
""
(replace-regexp-in-string
(rx bol "#+tblfm" (* (or any "\n")) eos)
""
(match-string 1 content))))))
(orgtbl-aggregate--insert-elisp-table
(orgtbl-aggregate--post-process
(orgtbl-aggregate--create-table-transposed
(orgtbl-aggregate--remove-cookie-lines
(orgtbl-aggregate-table-from-any-ref (plist-get params :table)))
(plist-get params :cols)
(plist-get params :cond))
post))
(orgtbl-aggregate--table-recalculate content formula)))
(defun orgtbl-aggregate--wizard-transpose-create-update (oldline expert)
"Update OLDLINE parameters by interactivly querying user.
OLDLINE is a plist containing parameter-value pairs.
Example: \\'(:table \"thetable\" :cols \"day month\" …)
OLDLINE is supposed to be extracted from an Org Mode block such as:
#+begin: transpose :table \"thetable\" :cols \"day month\" …
If (point) is not on such a line, OLDLINE is nil.
The function returns a plist which is an updated version of OLDLINE
amended by the user.
When EXPERT is nil, only basic parameters are queried.
Note that when an expert parameter was set prior to entering the wizard,
it is queried even when EXPERT is nil."
(let ((minibuffer-local-completion-map
(define-keymap :parent minibuffer-local-completion-map
"SPC" nil)) ;; allow inserting spaces
table headerlist header aggcols aggcond postprocess params)
(save-window-excursion
(setq table
(orgtbl-aggregate--wizard-query-table
(orgtbl-aggregate--plist-get-remove oldline :table)
expert))
(setq headerlist
(orgtbl-aggregate--get-header-table table))
(setq header
(mapconcat
(lambda (x) (format " ~%s~" x))
headerlist))
(orgtbl-aggregate--display-help :cols-tr header)
(setq aggcols
(replace-regexp-in-string
"\"" "'"
(read-string
"Target columns & formulas: "
(orgtbl-aggregate--merge-list-into-single-string
(orgtbl-aggregate--plist-get-remove oldline :cols))
'orgtbl-aggregate-history-cols)))
(setq aggcond (orgtbl-aggregate--plist-get-remove oldline :cond))
(when (or expert aggcond)
(orgtbl-aggregate--display-help :cond header)
(setq aggcond
(read-string
"Row filter (optional): "
aggcond
'orgtbl-aggregate-history-cols)))
(setq postprocess (orgtbl-aggregate--plist-get-remove oldline :post))
(when (or expert postprocess)
(orgtbl-aggregate--display-help :post)
(setq postprocess
(read-string
"Post process (optional): "
postprocess
'orgtbl-aggregate-history-cols)))
)
(setq params
(list
:name "transpose"
:table table
:cols aggcols))
(unless (eq (length aggcond) 0)
(nconc params `(:cond ,(read aggcond))))
(unless (eq (length postprocess) 0)
(nconc params `(:post ,postprocess)))
;; recover parameters not taken into account by the wizard
(cl-loop
for pair on oldline
if (car pair)
do (nconc params `(,(car pair) ,(cadr pair)))
do (setq pair (cdr pair)))
params))
;;;###autoload
(defun orgtbl-aggregate-insert-dblock-transpose (&optional expert)
"Wizard to interactively insert a transpose dynamic block."
(interactive "P")
(let* ((oldline (orgtbl-aggregate--parse-header-arguments "transpose"))
(params
(save-excursion (orgtbl-aggregate--wizard-transpose-create-update oldline expert)))
tblfm)
(when oldline
(org-mark-element)
(setq tblfm
(orgtbl-aggregate--recover-TBLFM
(buffer-substring-no-properties
(region-beginning) (1- (region-end)))))
(delete-region (region-beginning) (1- (region-end))))
(org-create-dblock params)
(when tblfm
(forward-line 1)
(insert "\n" tblfm)
(forward-line -2))
(org-update-dblock)))
;; [bazilo synchronize orgtbl-αggregate & orgtbl-joιn
;; Insert a dynamic bloc with the C-c C-x x dispatcher
;; and activate TAB on #+begin: aggregate ...
;;;###autoload
(eval-after-load 'org
'(progn
;; org-dynamic-block-define found in Emacs 27.1
(org-dynamic-block-define "aggregate" #'orgtbl-aggregate-insert-dblock-aggregate)
(org-dynamic-block-define "transpose" #'orgtbl-aggregate-insert-dblock-transpose)))
;; This hook will only work if orgtbl-aggregate is loaded,
;; thus the eval-after-load 'orgtbl-aggregate
;; We do not want this hook to be added to Org Mode if orgtbl-aggregate
;; is not used, thus the eval-after-load 'orgtbl-aggregate
;;;###autoload
(eval-after-load 'orgtbl-aggregate
'(add-hook 'org-cycle-tab-first-hook #'orgtbl-aggregate-dispatch-TAB))
;; bazilo]
(provide 'orgtbl-aggregate)
;;; orgtbl-aggregate.el ends here
================================================
FILE: orgtbl-aggregate.info
================================================
This is orgtbl-aggregate.info, produced by makeinfo version 6.8 from
orgtbl-aggregate.texi.
INFO-DIR-SECTION Emacs
START-INFO-DIR-ENTRY
* Orgtbl Aggregate: (orgtbl-aggregate). Create an aggregated Org table from another one
END-INFO-DIR-ENTRY
INFO-DIR-SECTION Misc
START-INFO-DIR-ENTRY
* (orgtbl-aggregate). Aggregate Values in a Table.
END-INFO-DIR-ENTRY
File: orgtbl-aggregate.info, Node: Top, Next: New, Up: (dir)
Aggregate Values in a Table
***************************
Aggregating a table is creating a new table by computing sums, averages,
and so on, out of material from the first table.
* Menu:
* New::
* Examples::
* Stop reading here! 80/20::
* Equivalent in SQL, R, Datamash, el-tblfn, Awk, C++: Equivalent in SQL R Datamash el-tblfn Awk C++.
* Wizards::
* Thecols parameter: The cols parameter.
* Column names::
* Formatters::
* Sorting::
* hlines in the output table::
* Cells processing::
* Wide variety of inputs::
* Post-processing::
* Pull & Push::
* Debugging::
* Tricks::
* Installation::
* Authors, contributors: Authors contributors.
* Changes::
* GPL 3 License::
— The Detailed Node Listing —
Examples
* A very simple example::
* Demonstrate sum and average computing::
* Example without days::
* Example of counting each combination::
Stop reading here! 80/20
* Name your input table::
* Create an aggregation block::
* Refresh the aggregation::
Equivalent in SQL, R, Datamash, el-tblfn, Awk, C++
* SQL equivalent::
* R equivalent::
* Datamash equivalent::
* el-tblfn::
* Awk equivalent::
* C++ equivalent::
Wizards
* Guiding (traditional) wizard::
* Experimental free form wizard::
The :cols parameter
* Names of input columns::
* Grouping specifications incols: Grouping specifications in cols.
* The hline column::
* The @# column::
* Aggregation formulas incols: Aggregation formulas in cols.
* Correlation of two columns::
* (Almost) any expression can be specified: [Almost) any expression can be specified.
Column names
* Input table with or without a header::
* Column names of the input table::
* Multiple lines header::
* Custom column names::
Formatters
* Org Mode compatible formatters::
* Debugging formatters::
* Discarding an output column::
Sorting
* Example with one sorting column::
* Several sorting columns::
hlines in the output table
* Output hlines depends on sorting columns::
* Example with hline 2::
Cells processing
* Where Calc interpretation happens?::
* Dates::
* Durations::
* Empty and malformed input cells::
* Symbolic computation::
* Intervals::
* Error or precision forms::
Wide variety of inputs
* Standard Org Mode input::
* Virtual input table from Babel::
* An Org ID::
* CSV input::
* JSON input::
* Input slicing::
* Thecond filter: The cond filter.
* Virtual input columns::
Post-processing
* Spreadsheet formulas::
* Algorithm post processing::
* Grand total::
* Chaining::
Pull & Push
* Pull mode::
* Push mode::
* Pull or push ?::
Debugging
* Seeing the $ forms::
* Seeing Calc formulas before evaluation::
* Seeing Lisp internal form of Calc formulas::
* Example of debugging vsum(nn^2)::
* Summary of debugging formatters::
Tricks
* Sorting: Sorting (1).
* A few lowest or highest values::
* Span of values::
* No aggregation::
File: orgtbl-aggregate.info, Node: New, Next: Examples, Prev: Top, Up: Top
1 New
*****
Transpose-babel blocks now handle ‘@#’ and ‘hline’ special columns.
‘@#’ is the input table row number. ‘hline’ is the block number between
two horizontal lines where the current row is located.
And by the way, yes, ‘orgtbl-aggregate’ comes with ‘orgtbl-transpose’
as a bonus. It flips rows with columns.
File: orgtbl-aggregate.info, Node: Examples, Next: Stop reading here! 80/20, Prev: New, Up: Top
2 Examples
**********
* Menu:
* A very simple example::
* Demonstrate sum and average computing::
* Example without days::
* Example of counting each combination::
File: orgtbl-aggregate.info, Node: A very simple example, Next: Demonstrate sum and average computing, Up: Examples
2.1 A very simple example
=========================
We have a table of activities and quantities (whatever they are) over
several days.
#+name: original
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
To begin with we want to gather all colors and count how many times
they appear. We are interested only in the second column named ‘Color’
First we give a name to the table through the ‘#+NAME:’ or
‘#+TBLNAME:’ tags, just above the table. Then we create a _dynamic
block_ to receive the aggregation:
desired output columns╶──────────────────────────╮
the input table╶──────────────╮ │
type of processing╶─╮ │ │
╭───────╯ │ │
▼ ╭───┴────╮ ╭─────┴───────╮
#+begin: aggregate :table "original" :cols "Color count()"
#+end:
Now typing ‘C-c C-c’ in the dynamic block counts the colors in the
original table:
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "Color count()"
| Color | count() |
|-------+---------|
| Red | 7 |
| Blue | 7 |
#+end:
OrgAggregate found two colors, ‘Red’ and ‘Blue’. It found 7
occurrences for each.
File: orgtbl-aggregate.info, Node: Demonstrate sum and average computing, Next: Example without days, Prev: A very simple example, Up: Examples
2.2 Demonstrate sum and average computing
=========================================
Now we want to aggregate this table for each day (because several rows
exist for each day). We want the average value of the ‘Level’ column
for each day, and the sum of the ‘Quantity’ column. We write down the
block specifying that (later we will see how to automate the creation of
such a block with a *note wizard: Wizards.):
sum aggregation╶─────────────────────────────────────────────────╮
average aggregation╶───────────────────────────────╮ │
key grouping column╶───────────────────────╮ │ │
╭┴╮ ╭────┴─────╮ ╭─────┴──────╮
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
#+end
Typing ‘C-c C-c’ in the dynamic block computes the aggregation:
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
╰┬╯ ╰────┬─────╯ ╰──────┬─────╯
╭───────────────────────────────────────╯ │ │
│ ╭───────────────────────────────╯ │
│ │ ╭──────────────────────────────╯
╭┴╮ ╭────┴─────╮ ╭─────┴──────╮
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+end
The source table is not changed in any way.
To get this result, we specified columns in this way, after the
‘:cols’ parameter:
• ‘Day’ : we got the same column as in the source table, except
entries are not duplicated. Here ‘Day’ acts as a _key grouping
column_. We may specify as many key columns as we want just by
naming them. We get only one aggregated row for each different
combination of values of key grouping columns.
• ‘vmean(Level)’ : this instructs OrgAggregate to compute the average
of values found in the ‘Level’ column, grouped by the same ‘Day’.
• ‘vsum(Quantity)’: OrgAggregate computes the sum of values found in
the ‘Quantity’ column, one sum for each ‘Day’.
File: orgtbl-aggregate.info, Node: Example without days, Next: Example of counting each combination, Prev: Demonstrate sum and average computing, Up: Examples
2.3 Example without days
========================
Maybe we are just interested in the sum of ‘Quantities’, regardless of
‘Days’. We just type:
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "vsum(Quantity)"
╰─────┬──────╯
╭──────────────────────────────────────────╯
╭────┴───────╮
| vsum(Quantity) |
|----------------|
| 218 |
#+end
File: orgtbl-aggregate.info, Node: Example of counting each combination, Prev: Example without days, Up: Examples
2.4 Example of counting each combination
========================================
we may want to count the number of rows for each combination of ‘Day’
and ‘Color’:
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+BEGIN: aggregate :table "original" :cols "count() Day Color"
╰──┬──╯ ╰┬╯ ╰─┬─╯
╭─────────────────────────────────────────╯ │ │
│ ╭───────────────────────────────────────╯ │
│ │ ╭───────────────────────────────╯
╭──┴──╮ ╭┴╮ ╭─┴─╮
| count() | Day | Color |
|---------+-----------+-------|
| 1 | Monday | Red |
| 1 | Monday | Blue |
| 2 | Tuesday | Red |
| 1 | Tuesday | Blue |
| 1 | Wednesday | Red |
| 2 | Wednesday | Blue |
| 3 | Thursday | Red |
| 3 | Friday | Blue |
#+END
If we want to get measurements for ‘Colors’ rather than ‘Days’, we
type:
C-c C-c here to refresh╶╮
╭──────────╯
▼
#+begin: aggregate :table "original" :cols "Color vmean(Level) vsum(Quantity)"
╰─┬─╯ ╰────┬─────╯ ╰─────┬──────╯
╭─────────────────────────────────────────╯ │ │
│ ╭──────────────────────────────────────╯ │
│ │ ╭────────────────────────────────────╯
╭─┴─╮ ╭────┴─────╮ ╭─────┴──────╮
| Color | vmean(Level) | vsum(Quantity) |
|-------+---------------+----------------|
| Red | 40.2857142857 | 144 |
| Blue | 15.5714285714 | 74 |
#+end
File: orgtbl-aggregate.info, Node: Stop reading here! 80/20, Next: Equivalent in SQL R Datamash el-tblfn Awk C++, Prev: Examples, Up: Top
3 Stop reading here! 80/20
**************************
If you managed to get here, you are at 80/20 (thanks Pareto!). You
grasped only 20% of the OrgAggregate features, but those 20% cover 80%
of the use cases.
To summarize the 20%:
* Menu:
* Name your input table::
* Create an aggregation block::
* Refresh the aggregation::
File: orgtbl-aggregate.info, Node: Name your input table, Next: Create an aggregation block, Up: Stop reading here! 80/20
3.1 Name your input table
=========================
• Select one of your Org table, and be ready to aggregate values from
it right in the same file.
• Give a name to your table with a special line just above it.
name here╶───╮
╭────╯
▼
#+name: original
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
…
File: orgtbl-aggregate.info, Node: Create an aggregation block, Next: Refresh the aggregation, Prev: Name your input table, Up: Stop reading here! 80/20
3.2 Create an aggregation block
===============================
#+begin: aggregate
#+end:
• *Input:* specify a ‘:table’ parameter.
• *Output:* specify the desired output columns with the ‘:cols’
parameter.
output╶───────────────────────────────────────────╮
input╶───────────────────────╮ │
│ │
╭──┴───╮ ╭────────┴─────────╮
#+begin: aggregate :table original :cols "Day vsum(Quantity)"
#+end:
File: orgtbl-aggregate.info, Node: Refresh the aggregation, Prev: Create an aggregation block, Up: Stop reading here! 80/20
3.3 Refresh the aggregation
===========================
• Type ‘C-c C-c’ on the ‘#+begin:’ line now and whenever you want to
refresh the aggregation.
C-c C-c here╶─╮
│
▼
#+begin: aggregate :table original :cols "Day vsum(Quantity)"
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Tuesday | 45 |
…
#+end:
File: orgtbl-aggregate.info, Node: Equivalent in SQL R Datamash el-tblfn Awk C++, Next: Wizards, Prev: Stop reading here! 80/20, Up: Top
4 Equivalent in SQL, R, Datamash, el-tblfn, Awk, C++
****************************************************
Aggregation is a widely used method to get insights in tabular data.
Use whatever environment best suits your needs.
OrgAggregate is great when you want to output and Org Mode table.
Also, OrgAggregate has no dependency other than Emacs, not even other
Lisp packages.
* Menu:
* SQL equivalent::
* R equivalent::
* Datamash equivalent::
* el-tblfn::
* Awk equivalent::
* C++ equivalent::
File: orgtbl-aggregate.info, Node: SQL equivalent, Next: R equivalent, Up: Equivalent in SQL R Datamash el-tblfn Awk C++
4.1 SQL equivalent
==================
If you are familiar with SQL, you would get a similar result with the
‘GROUP BY’ statement:
select Day, mean(Level), sum(Quantity)
from original
group by Day;
File: orgtbl-aggregate.info, Node: R equivalent, Next: Datamash equivalent, Prev: SQL equivalent, Up: Equivalent in SQL R Datamash el-tblfn Awk C++
4.2 R equivalent
================
If you are familiar with the R statistical language, you would get a
similar result with ‘factor’ and ‘aggregate’ functions:
original <- the table as a data.frame
day_factor <- factor(original$Day)
aggregate (original$Level , list(Day=day_factor), mean)
aggregate (original$Quantity, list(Day=day_factor), sum )
File: orgtbl-aggregate.info, Node: Datamash equivalent, Next: el-tblfn, Prev: R equivalent, Up: Equivalent in SQL R Datamash el-tblfn Awk C++
4.3 Datamash equivalent
=======================
The command-line Datamash software operates on CSV files and can achieve
a similar result:
datamash -H -g Day mean Level sum Quantity . Example:
#+begin_src elisp :var original=original :colnames no :hlines yes
(require 'tblfn)
(thread-first
(tblfn-aggregate original "Quantity" "Total"))
#+end_src
It is not clear how el-tblfn can compute the ‘mean’ (or other
aggregating functions) on the ‘Level’ column. Would the author want to
complete the example?
File: orgtbl-aggregate.info, Node: Awk equivalent, Next: C++ equivalent, Prev: el-tblfn, Up: Equivalent in SQL R Datamash el-tblfn Awk C++
4.5 Awk equivalent
==================
Awk is a line-oriented filter which has been arround in Unix for
decades. Here we use the standard Org Mode support for Awk.
#+begin_src awk :stdin original
NR>1 {
Day =$1
Color =$2
Level =$3
Quantity =$4
SumLevel[Day] += Level
SumQuantity[Day] += Quantity
Count[Day] ++
}
END {
for (d in SumQuantity) {
printf "%s %s %s\n", d, SumLevel[d]/Count[d], SumQuantity[d]
}
}
#+end_src
File: orgtbl-aggregate.info, Node: C++ equivalent, Prev: Awk equivalent, Up: Equivalent in SQL R Datamash el-tblfn Awk C++
4.6 C++ equivalent
==================
C++ has hash-maps in its standard template library. And Org Mode
provides support for C++ Babel blocks. Thus, it is quite
straigthforward to aggegrate in this language.
(Don’t forget to customize ‘org-src-lang-modes’ to activate C++
support in Org Mode).
#+begin_src C++ :var original=original :includes '( )
using namespace std;
unordered_map SumLevel;
unordered_map SumQuantity;
unordered_map Count;
for (auto row : original) {
auto Day = row[0];
auto Color = row[1];
auto Level = stod(row[2]);
auto Quantity = stod(row[3]);
SumLevel[Day] += Level;
SumQuantity[Day] += Quantity;
Count[Day] ++;
}
for (auto it : SumQuantity)
cout<’ so that it does not appear in the output.
invisible╶─────────────────────────────────────────╮
sorted numerically decreasing╶───────────────────╮ │
row numbers of input table╶──────────────────╮ │ │
│ │ │
▼ ▼ ▼
#+begin: aggregate :table "original" :cols "Day Color Level Quantity @#;^N;<>"
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Friday | Blue | 11 | 9 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 7 | 5 |
| Thursday | Red | 49 | 30 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 39 | 24 |
| Wednesday | Blue | 15 | 15 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Red | 27 | 23 |
| Tuesday | Blue | 33 | 18 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Red | 51 | 12 |
| Monday | Blue | 25 | 3 |
| Monday | Red | 30 | 11 |
#+end:
File: orgtbl-aggregate.info, Node: Aggregation formulas in cols, Next: Correlation of two columns, Prev: The @# column, Up: The cols parameter
6.5 Aggregation formulas in :cols
=================================
Aggregation formulas are applied for each of the groupings, on the
specified columns.
We saw examples with ‘sum’, ‘mean’, ‘count’ aggregations. There are
many other aggregations. They are based on functions provided by Calc
(Calc is the powerful Emacs calculator):
• ‘count()’ or ‘vcount()’
• in Calc: ‘`u #' (`calc-vector-count') [`vcount'])’
• gives the number of elements in the group being aggregated;
this function may or may not take a column parameter; with a
parameter, empty cells are not counted (except with the ‘E’
modifier)..
• ‘sum(X)’ or ‘vsum(X)’
• in Calc: ‘`u +' (`calc-vector-sum') [`vsum']’
• computes the sum of elements being aggregated
• ‘cnorm(X)’
• in Calc: ‘`v N' (calc-cnorm') [`cnorm']’
• like ‘vsum(X)’, compute the sum of values, but first replacing
negative values by their opposite
• ‘max(X)’ or ‘vmax(X)’
• in Calc: ‘`u X' (`calc-vector-max') [`vmax']’
• gives the largest of the elements being aggregated
• ‘min(X)’ or ‘vmin(X)’
• in Calc: ‘`u N' (`calc-vector-min') [`vmin']’
• gives the smallest of the elements being aggregated
• ‘span(X)’ or ‘vspan(X)’
• in Calc: ‘`v :' (`calc-set-span') [`vspan']’
• summarizes values to be aggregated into an interval
‘[MIN..MAX]’ where ‘MIN’ and ‘MAX’ are the minimal and maximal
values to be aggregated
• ‘rnorm(X)’
• in Calc: ‘`v n' (`calc-rnorm) [`rnorm']’
• like ‘vmax(X)’, gives the maximum of values, but first
replacing negative values by their opposite
• ‘mean(X)’ or ‘vmean(X)’
• in Calc: ‘`u M' (`calc-vector-mean') [`vmean']’
• computes the average (arithmetic mean) of elements being
aggregated
• ‘meane(X)’ or ‘vmeane(X)’
• in Calc: ‘`I u M' (`calc-vector-mean-error') [`vmeane']’
• computes the average (as mean) along with the estimated error
of elements being aggregated
• ‘median(X)’ or ‘vmedian(X)’
• in Calc: ‘`H u M' (`calc-vector-median') [`vmedian']’
• computes the median of elements being aggregated, by taking
the middle element after sorting them
• ‘hmean(X)’ or ‘vhmean(X)’
• in Calc: ‘`H I u M' (`calc-vector-harmonic-mean') [`vhmean']’
• computes the harmonic mean of elements being aggregated
• ‘gmean(X)’ or ‘vgmean(X)’
• in Calc: ‘`u G' (`calc-vector-geometric-mean') [`vgmean']’
• computes the geometric mean of elements being aggregated
• ‘sdev(X)’ or ‘vsdev(X)’
• in Calc: ‘`u S' (`calc-vector-sdev') [`vsdev']’
• computes the standard deviation of elements being aggregated
• ‘psdev(X)’ or ‘vpsdev(X)’
• in Calc: ‘`I u S' (`calc-vector-pop-sdev') [`vpsdev']’
• computes the population standard deviation (divide by N
instead of N-1)
• ‘var(X)’ or ‘vvar(X)’
• in Calc: ‘`H u S' (`calc-vector-variance') [`vvar']’
• computes the variance of elements being aggregated
• ‘pvar(X)’ or ‘vpvar(X)’
• in Calc: ‘`H u S' (`calc-vector-variance') [`vpvar']’
• computes the population variance of elements being aggregated
• ‘pcov(X,Y)’ or ‘vpcov(X,Y)’
• in Calc: ‘`I u C' (`calc-vector-pop-covariance') [`vpcov']’
• computes the population covariance of elements being
aggregated from two columns (divides by N)
• ‘cov(X,Y)’ or ‘vcov(X,Y)’
• in Calc: ‘`u C' (`calc-vector-covariance') [`vcov']’
• computes the sample covariance of elements being aggregated
from two columns (divides by N-1)
• ‘corr(X,Y)’ or ‘vcorr(X,Y)’
• in Calc: ‘`H u C' (`calc-vector-correlation') [`vcorr']’
• computes the linear correlation coefficient of elements being
aggregated in two columns
• ‘prod(X)’ or ‘vprod(X)’
• in Calc: ‘`u *' (`calc-vector-product') [`vprod']’
• computes the product of elements being aggregated
• ‘vlist(X)’ or ‘list(X)’
• gives the list of ‘X’ being aggregated, verbatim, without
aggregation.
• ‘(X)’ or ‘X’ in a formula
• returns the list of ‘X’ being aggregated, without aggregation,
passed through Calc interpretation.
• ‘sort(X)’
• in Calc: ‘`v S' (`calc-sort') [`sort']’
• sorts elements to be aggregated in ascending order; only works
on numerical values
• ‘rsort(X)’
• in Calc: ‘`I v S' (`calc-sort') [`sort']’
• sorts elements to be aggregated in descending order; only
works on numerical values
• ‘rev(X)’
• in Calc: ‘`' (`calc-reverse-vector') [`rev']’
• returns the list of values to be aggregated in reverse order
• ‘subvec(X,from)’, ‘subvec(X,from,to)’
• in Calc: ‘`v s' (`calcFunc-subvec') [`subvec']’
• extracts a sub-list from ‘X’ starting at ‘from’ and ending at
‘to’ excluded (or up to the end if ‘to’ is not given). The
first value is numbered ‘1’. So for instance ‘subvec(X,1,3)’
extracts the first two values
• ‘vmask(M,X)’
• in Calc: ‘`v m' (`calcFunc-vmask') [`vmask']’
• extracts a sub-list from ‘X’, keeping only values for which
corresponding values in ‘M’ (the mask) are not zero
• ‘head(X)’
• in Calc: ‘`v h' (`calc-head') [`head']’
• returns the first value to be aggregated
• ‘rtail(X)’
• in Calc: ‘`H I v h' (`calc-head') [`rtail']’
• returns the last value to be aggregated
• ‘find(X,val)’
• in Calc: ‘`v f' (`calc-vector-find') [`find']’
• returns the index of ‘val’ in the list of values to be
aggregated, or ‘0’ if ‘val’ is not found. Index starts from
‘1’
• ‘rdup(X)’
• in Calc: ‘`v +' (`calc-remove-duplicates') [`rdup']’
• remove duplicates from ‘X’ and returns remaining values sorted
in ascending order
• ‘grade(X)’
• in Calc: ‘`v G' (`calc-grade') [`grade']’
• returns a list of index of values to be aggregated: the index
of the lowest value, then the second lowest value, and so on
up to the index of the highest value. Indexes start from ‘1’
• ‘rgrade(X)’
• in Calc: ‘`I v G' (`calc-grade') [`rgrade']’
• Like ‘grade’ in reverse order
The aggregation functions may be written with or without a leading
‘v’. ‘sum’ and ‘vsum’ are equivalent. The ‘v’ form should be
preferred, as it is the one used in the Org table spreadsheet, and in
Calc. The non-v names may be dropped in the future.
File: orgtbl-aggregate.info, Node: Correlation of two columns, Next: [Almost) any expression can be specified, Prev: Aggregation formulas in cols, Up: The cols parameter
6.6 Correlation of two columns
==============================
Some aggregations work on two columns (rather than one column for
‘vsum()’, ‘vmean()’). Those aggregations are ‘vcov(,)’, ‘vpcov(,)’,
‘vcorr(,)’.
• ‘vcorr(,)’ computes the linear correlation between two columns.
• ‘vcov(,)’ and ‘vpcov(,)’ compute the covariance of two columns.
Example. We create a table where column ‘y’ is a noisy version of
column ‘x’:
#+tblname: noisydata
| bin | x | y |
|-------+----+---------|
| small | 1 | 10.454 |
| small | 2 | 21.856 |
| small | 3 | 30.678 |
| small | 4 | 41.392 |
| small | 5 | 51.554 |
| large | 6 | 61.824 |
| large | 7 | 71.538 |
| large | 8 | 80.476 |
| large | 9 | 90.066 |
| large | 10 | 101.070 |
| large | 11 | 111.748 |
| large | 12 | 121.084 |
#+tblfm: $3=$2*10+random(1000)/500;%.3f
#+BEGIN: aggregate :table noisydata :cols "bin vcorr(x,y) vcov(x,y) vpcov(x,y)"
| bin | vcorr(x,y) | vcov(x,y) | vpcov(x,y) |
|-------+----------------+---------------+---------------|
| small | 0.999459736649 | 25.434 | 20.3472 |
| large | 0.999542438688 | 46.4656666667 | 39.8277142857 |
#+END
We see that the correlation between ‘x’ and ‘y’ is very close to ‘1’,
meaning that both columns are correlated. Indeed they are, as the ‘y’
is computed from ‘x’ with the formula
y = 10*x + noise_between_0_and_2
File: orgtbl-aggregate.info, Node: [Almost) any expression can be specified, Prev: Correlation of two columns, Up: The cols parameter
6.7 (Almost) any expression can be specified
============================================
Virtually any Calc formula can be specified as an aggregation formula.
Single column name (as they appear in the header of the source table,
or in the form of ‘$1’, ‘$2’, ..., or the virtual columns ‘hline’ and
‘@#’) are key columns. Everything else is given to Calc, to be computed
as an aggregation.
For instance:
(3) ;; a constant
vmean(2*X+1) ;; aggregate an expression
exp(vmean(map(log,N))) ;; the exponential average
vsum((X-vmean(X))^2) ;; X-vmean(X) centers the sample on zero
Arguably, the first expression is useless, but legal. The
aggregation can be applied to a computed list of values. The result of
an aggregation can be further processed in a formula. An aggregation
can even be applied to an expression containing another aggregation.
In an expression, if a variable has the name of a column, then it is
replaced by a Calc vector containing values from this column.
The special expression ‘(C)’ (a column name within parenthesis)
yields a list of values to be aggregated from this column, except they
are not aggregated. Note that parenthesis are required, otherwise, ‘C’
would act as a key grouping column.
File: orgtbl-aggregate.info, Node: Column names, Next: Formatters, Prev: The cols parameter, Up: Top
7 Column names
**************
* Menu:
* Input table with or without a header::
* Column names of the input table::
* Multiple lines header::
* Custom column names::
File: orgtbl-aggregate.info, Node: Input table with or without a header, Next: Column names of the input table, Up: Column names
7.1 Input table with or without a header
========================================
The header of a table gives names to its columns. It is separated from
data with an horizontal line.
column name is "quantity" or "$2"╶╮
column name is "day" or "$1"╶╮ │
╭─────────────────────────╯ │
│ ╭───────────────╯
▼ ▼
| day | quantity |
|-----------+----------|
| monday | 12.3 |
| monday | 5.9 |
| thursday | 41.1 |
| wednesday | 16.8 |
In this example, the input columns may be referred to as ‘day’ and
‘quantity’.
Tables without a header are handled by OrgAggregate with _"dollar
names"_. Example of a table without a header:
column name is "$2"╶──╮
column name is "$1"╶╮ │
╭───────────────╯ │
│ ╭─╯
▼ ▼
| monday | 12.3 |
| monday | 5.9 |
| thursday | 41.1 |
| wednesday | 16.8 |
Then columns may be refereed to as ‘$1’ and ‘$2’.
File: orgtbl-aggregate.info, Node: Column names of the input table, Next: Multiple lines header, Prev: Input table with or without a header, Up: Column names
7.2 Column names of the input table
===================================
Column names are not necessarily alphanumeric words. They may contain
any characters, including spaces, quotes, +, -, whatever. They must not
extend on several lines thought.
Those names need to be protected with quotes (single or double
quotes) within formulas.
Examples:
• ‘:cols’ "‘mean('estimated value')’"
• ‘:cond (equal "true color" "Red")’
Quoting is not required for
• ASCII letters
• numbers
• underscore _, dollar $, dot .
• accented letters like à é
• Greek letters like α, Ω
• northern letters like ø
• Russian letters like й
• Esperanto letters like ŭ
• Japanese ideograms like 量
Note that in ‘:cond’ Lisp expression, only double quotes work. This
is because single quotes in Lisp have a very special meaning.
‘Ubuntu Mono’ font can be used for displaying aligned Japanese
characters, although not perfectly.
File: orgtbl-aggregate.info, Node: Multiple lines header, Next: Custom column names, Prev: Column names of the input table, Up: Column names
7.3 Multiple lines header
=========================
The header of the source table may be more than one row tall. Only the
first header row is used to match column names between the source table
and the ‘:cols’ specifications.
Best effort is made to propagate additional header rows to the
aggregated table. This happens when the aggregated column refers to a
single source column, either as a key column or a formula involving a
single column.
#+name: tall-header
╭──────────────────────────────────────────────╮
╭───┴──╮ │
| color | quantity | level | ╶╮ │
| | | <3> | ├─╴header is 3 rows tall │
| kolor | kiom | nivelo | ╶╯ │
|--------+----------+--------| ╭───────────╯───────────────╮
| yellow | 72 | 3 | │ only the first header row │
| green | 55 | 5 | │ is used in formulas │
| | | | ╰───────────╭───────────────╯
| orange | 80 | 2 | │
| yellow | 13 | 1 | │
╭───┴──╮
#+BEGIN: aggregate :table "tall-header" :cols "color vsum(quantity);'sum' count();'nb' vsum(quantity)/vmean(level);'leveled'"
| color | sum | nb | leveled | ╶╮
| | | | | ├──╮
| kolor | kiom | | | ╶╯ │ ╭────────────────────────╮
|--------+------+----+---------| │ │ best attempt to recover│
| yellow | 85 | 2 | 42.5 | ╰──┤ the three header rows │
| green | 55 | 1 | 11 | │ in the output │
| orange | 80 | 1 | 40 | ╰────────────────────────╯
#+END:
Note that the last aggregated column has just ‘leveled’ in its
header. This is because this column refers to more than one source
columns, namely ‘quantity’ and ‘level’.
Note that in this example, there are formatting cookies:
<> <7>
Data rows containing at least one cookie are ignored. They are not
ignored in the header.
File: orgtbl-aggregate.info, Node: Custom column names, Prev: Multiple lines header, Up: Column names
7.4 Custom column names
=======================
In this example, output column have names which are difficult to handle:
#+BEGIN: aggregate :table original :cols "Day vmean(Level*2) vsum(Quantity^2)"
╰─────┬──────╯ ╰──────┬───────╯
╭───────────────────────────────╯ │
│ ╭─────────────────────────────╯
╭─────┴──────╮ ╭──────┴───────╮
| Day | vmean(Level*2) | vsum(Quantity^2) |
|-----------+----------------+------------------|
| Monday | 55 | 130 |
| Tuesday | 86 | 693 |
| Wednesday | 36 | 1010 |
| Thursday | 86 | 2317 |
| Friday | 16 | 170 |
#+END
We can give them custom names with the ‘;'custom name'’ decoration:
#+BEGIN: aggregate :table original :cols "Day vmean(Level*2);'mean2' vsum(Quantity^2);'sum_squares'"
╰──┬──╯ ╰─────┬─────╯
╭───────────────────────────────────────────────╯ │
│ ╭─────────────────────────────────────────────────────────────────╯
╭─┴─╮ ╭───┴─────╮
| Day | mean2 | sum_squares |
|-----------+-------+-------------|
| Monday | 55 | 130 |
| Tuesday | 86 | 693 |
| Wednesday | 36 | 1010 |
| Thursday | 86 | 2317 |
| Friday | 16 | 170 |
#+END
Decorators are optional.
File: orgtbl-aggregate.info, Node: Formatters, Next: Sorting, Prev: Column names, Up: Top
8 Formatters
************
* Menu:
* Org Mode compatible formatters::
* Debugging formatters::
* Discarding an output column::
File: orgtbl-aggregate.info, Node: Org Mode compatible formatters, Next: Debugging formatters, Up: Formatters
8.1 Org Mode compatible formatters
==================================
An expression may optionally be followed by modifiers and formatters,
after a semicolon. Examples:
vsum(X);p20 ;; increase Calc internal precision to 20 digits
vsum(X);f3 ;; output the result with 3 digits after the decimal dot
vsum(X);%.3f ;; output the result with 3 digits after the decimal dot
The modifiers and formatters are fully compatible with those of the
Org Mode spreadsheet.
• ‘p12’ change the precision to 12 decimal digits.
• ‘n7’ output as floating point number with 7 decimal digits.
• ‘f4’ output number with 4 decimal places after dot.
• ‘s5’ output number in "scientific" mode (with exponent of 10) with
5 decimal digits.
• ‘e6’ output number in "engineering" mode (with exponent of 10
multiple of 3) with 6 decimal digits.
• ‘t’ output duration in decimal hours; input is supposed to be
either a duration like ‘"2:37"’ meaning 2 hours and 37 minutes, or
a number of seconds like ‘"1234’" which is approximately ‘0.34’
hours. The output is controlled by the
‘org-table-duration-custom-format’ variable.
• ‘T’ output duration in an hours-minutes-seconds format like
‘"01:20:34"’ meaning 1 hour, 20 minutes, and 34 seconds.
• ‘U’ like ‘t’, but disregard the ‘org-table-duration-custom-format’
variable and use ‘hh:mm’ in place.
• ‘N’ output number: remove any non-numeric output.
• ‘E’ keep empty input cells. The result is often ‘nan’. Without
‘E’, empty input cells are ignored as if they did not exist.
• ‘D’ angles are in degrees.
• ‘R’ angles are in radians.
• ‘F’ output is presented as a fraction of integers if it actually
is. The format is the Calc one, for example ‘"2:3"’ means ‘2/3’.
• ‘S’ symbolic mode. When an input cell is, for instance ‘sqrt(2)’,
it it kept as-is rather than being replaced by ‘1.41421’.
File: orgtbl-aggregate.info, Node: Debugging formatters, Next: Discarding an output column, Prev: Org Mode compatible formatters, Up: Formatters
8.2 Debugging formatters
========================
Additionally, a few formatters are dedicated to debugging:
• ‘c’ output the Calc expression before substitution by actual input
cells values.
• ‘q’ output the Lisp expression before substitution by actual input
cells values.
• ‘C’ output the Calc expression before it gets simplified and
folded.
• ‘Q’ output the Lisp expression before it gets simplified and
folded.
See *note Debugging:: for a detailed explanation.
File: orgtbl-aggregate.info, Node: Discarding an output column, Prev: Debugging formatters, Up: Formatters
8.3 Discarding an output column
===============================
Why would anyone specify a column just to discard it in the output? For
its side effects. For sorting the output table or for adding hlines to
it.
To discard a column, add a ‘;<>’ modifier to the column description.
This syntax is reminiscent of the ‘’ cookies in Org Mode tables,
which instructs to shorten a column width to only ‘n’ characters.
In this example, input hlines create a ‘hline’ column which is used
to add hlines to the output. Then this ‘hline’ column is discarded with
‘<>’.
invisible╶────────────────────────────────────────────╮
sorted numerically increasing╶──────────────────────╮ │
│ │
▼ ▼
#+BEGIN: aggregate :table "withhline" :cols "hline;^n;<> cölØr vsum(vâluε)" :hline 1
| cölØr | vsum(vâluε) |
|--------+-------------|
| Red | 7.4 |
| Yellow | 9.1 |
|--------+-------------|
| Blue | 15.7 |
| Yellow | 5.4 |
|--------+-------------|
| Blue | 4.9 |
| Red | 3.9 |
| Yellow | 9. |
|--------+-------------|
| Red | 1.1 |
| Yellow | 3.4 |
#+END:
Here is an example where rows are sorted on the ‘cölØr’ column, but
without displaying this column:
invisible╶────────────────────────────────────────────╮
sorted alphabetically╶──────────────────────────────╮ │
│ │
▼ ▼
#+BEGIN: aggregate :table "withhline" :cols "cölØr;^a;<> vâluε;^n" :hline 1
| vâluε | ▲
|-------| │
| 4.9 | within the same cölØr bucket, │
| 7.0 | sort vâluε numerically╶─────────────────────╯
| 8.7 |
|-------|
| 1.1 |
| 1.3 |
| 2.6 |
| 3.5 |
| 3.9 |
|-------|
| 2.4 |
| 3.4 |
| 5.4 |
| 6.6 |
| 9.1 |
#+END:
File: orgtbl-aggregate.info, Node: Sorting, Next: hlines in the output table, Prev: Formatters, Up: Top
9 Sorting
*********
* Menu:
* Example with one sorting column::
* Several sorting columns::
File: orgtbl-aggregate.info, Node: Example with one sorting column, Next: Several sorting columns, Up: Sorting
9.1 Example with one sorting column
===================================
In this example, the output table is sorted numerically on its second
column (look at the ‘^n’ specification):
#+begin: aggregate :table "original" :cols "Day vsum(Quantity);^n"
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Friday | 22 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
#+end:
By default, no sorting is done. The output rows follows the ordering
of the input rows.
Any column specification in the ‘:cols’ parameter may be followed by
a semicolon and caret characters, and an ordering.
The specification for the ordering are the same as in Org Mode:
• ‘a’: ascending alphabetical sort
• ‘A’: descending alphabetical sort
• ‘n’: ascending numerical sort
• ‘N’: descending numerical sort
• ‘t’: ascending date, time, or duration sort
• ‘T’: descending date, time, or duration sort
• ‘f’ & ‘F’ specifications are not (yet) implemented
File: orgtbl-aggregate.info, Node: Several sorting columns, Prev: Example with one sorting column, Up: Sorting
9.2 Several sorting columns
===========================
The rows of the resulting table may be sorted on any combination of its
columns.
Several columns may get a sorting specification. The major column is
used for sorting. Only when two rows are equal regarding the major
column, the second major column is compared. And if the two rows are
still equal on this second column, the third is used, and so on.
The first sorted column in the ‘:cols’ parameter is the major one.
To declare another one as the major, follow it with a number, for
instance ‘1’. Columns without a number are minor ones.
Example:
:cols "AAA;^a BBB;^N2 CCC DDD;^t1"
╭────╮ ▲╷ ▲▲ ▲▲ ╭──────────────╮
│sort╰──────╯╰─╮ ││ │╰─╯first priority│
│alphabetically│ ││ ╰──╮sort by date │
│third priority│ ││ ╰──────────────╯
╰──────────────╯ ││ ╭────────────────╮
│╰──╯second priority │
╰───╮sort numerically│
│decreasing │
╰────────────────╯
• Column ‘DDD’ is sorted in ascending dates or times (‘t’
specification). It is the major sorting column (because of its ‘1’
numbering).
• Column ‘BBB’ sorts rows which compare equal on column ‘DDD’
(because of its ‘2’ numbering). This column is assumed to contain
numerical values, and it is sorted in descending order (‘N’
specification).
• Column ‘AAA’ is used to sort rows which compare equal regarding
‘DDD’ and ‘BBB’. It is sorted in ascending alphabetical order (‘a’
specification).
Both a format and a sorting instruction may be given. Example:
:cols "EXPR:f3:^n"
The ‘EXPR’ column is
• formatted with 3 digits after dot (‘f3’)
• sorted numerically in ascending order (‘^n’).
File: orgtbl-aggregate.info, Node: hlines in the output table, Next: Cells processing, Prev: Sorting, Up: Top
10 hlines in the output table
*****************************
The ‘:hline N’ parameter controls horizontal lines in the output table.
It may or may not be related to horizontal lines in the input.
* Menu:
* Output hlines depends on sorting columns::
* Example with hline 2::
File: orgtbl-aggregate.info, Node: Output hlines depends on sorting columns, Next: Example with hline 2, Up: hlines in the output table
10.1 Output hlines depends on sorting columns
=============================================
Example of an input table:
#+name: withouthline
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
Horizontal lines appear on the sorted column, which in this example
is the ‘cölØr’ column.
We require output hlines with ‘:hline 1’. The ‘1’ value here says
that only one sorted column should be considered when drawing output
horizontal lines. A value of ‘2’ would mean to consider two sorted
columns.
Horizontal lines will separate blocks of identical ‘cölØr’ rows:
#+BEGIN: aggregate :table "withouthline" :cols "cölØr;^a vâluε 'ra;han'" :hline 1
| cölØr | vâluε | 'ra;han' | ╰─┬─╯ ▲ ╰─┬─╯
|--------+-------+----------| │ │ │
| Blue | 8.7 | 52 |╶╮ │ │╭──────╮ ╭─────────┴─────╮
| Blue | 7.0 | 29 | ├─╴Blue bucket │ ╰╯sort │ │ separate │
| Blue | 4.9 | 64 |╶╯ ╰─────╮cölØr │ │ cölØr buckets │
|--------+-------+----------| ◀────────────────╮ ╰──────╯ │ with hlines │
| Red | 1.3 | 41 |╶╮ │ ╰──┬┬───────────╯
| Red | 3.5 | 35 | │ ╰─────────────────────╯│
| Red | 2.6 | 84 | ├─╴Red bucket │
| Red | 3.9 | 51 | │ │
| Red | 1.1 | 58 |╶╯ │
|--------+-------+----------| ◀───────────────────────────────────────╯
| Yellow | 9.1 | 95 |╶╮
| Yellow | 5.4 | 17 | │
| Yellow | 2.4 | 55 | ├─╴Yellow bucket
| Yellow | 6.6 | 34 | │
| Yellow | 3.4 | 51 |╶╯
#+END:
File: orgtbl-aggregate.info, Node: Example with hline 2, Prev: Output hlines depends on sorting columns, Up: hlines in the output table
10.2 Example with hline 2
=========================
In the following example, we specify ‘:hline 2’.
First, the input table now have horizontal lines. We want to
propagate them to the output.
#+name: withhline
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
|--------+-------+--------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
|--------+-------+--------|
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+--------|
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
The two sorted columns are ‘hline’ and ‘cölØr’. Therefore output
horizontal lines separate blocks of identical ‘hline’ and ‘cölØr’:
#+begin: aggregate :table "withhline" :cols "hline;^n cölØr;^a vâluε 'ra;han'" :hline 2
| hline | cölØr | vâluε | 'ra;han' | ▲ ▲ ▲
|-------+--------+-------+----------| │ │ │
| 0 | Red | 1.3 | 41 | ╭──────╯ ╰───────────╮ │
| 0 | Red | 3.5 | 35 | │ two sorted output columns │ │
| 0 | Red | 2.6 | 84 | ╰───────────────────────────╯ │
|-------+--------+-------+----------| │
| 0 | Yellow | 9.1 | 95 | │
|-------+--------+-------+----------| ╭─────────────────────────────╮ │
| 1 | Blue | 8.7 | 52 | │ 2 means: create hlines ├─────╯
| 1 | Blue | 7.0 | 29 | │ for buckets and sub-buckets │
|-------+--------+-------+----------| ╰─────────────────────────────╯
| 1 | Yellow | 5.4 | 17 |
|-------+--------+-------+----------|╶─╮
| 2 | Blue | 4.9 | 64 | │ ╭──────────────────────────╮
|-------+--------+-------+----------| ╶┤ │ bucket hline=2 │
| 2 | Red | 3.9 | 51 | ├───┤ divided in 3 sub-buckets │
|-------+--------+-------+----------| ╶┤ │ Blue, Red, Yellow │
| 2 | Yellow | 2.4 | 55 | │ ╰──────────────────────────╯
| 2 | Yellow | 6.6 | 34 | │
|-------+--------+-------+----------|╶─╯
| 3 | Red | 1.1 | 58 |
|-------+--------+-------+----------|
| 3 | Yellow | 3.4 | 51 |
#+end:
And the ‘hline’ column may be discarded (but its side effect
remains). To do so use the ‘;<>’ specifier:
#+begin: aggregate :table "withhline" :cols "hline;^n;<> cölØr;^a vâluε 'ra;han'" :hline 2
| cölØr | vâluε | 'ra;han' |
|--------+-------+----------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Red | 2.6 | 84 |
|--------+-------+----------|
| Yellow | 9.1 | 95 |
|--------+-------+----------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
|--------+-------+----------|
| Yellow | 5.4 | 17 |
|--------+-------+----------|
| Blue | 4.9 | 64 |
|--------+-------+----------|
| Red | 3.9 | 51 |
|--------+-------+----------|
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+----------|
| Red | 1.1 | 58 |
|--------+-------+----------|
| Yellow | 3.4 | 51 |
#+end:
The ‘:hline’ parameter accepts a number:
• ‘:hline 0’, ‘:hline no’, ‘:hline nil’, or no ‘:hline’ mean that
there will be no hlines in the output.
• ‘:hline 1’, ‘:hline yes’, ‘:hline t’ mean that hlines will separate
blocks of identical rows regarding the major sorted column. In
case no column is sorted, then output hlines will reflect input
ones.
• ‘:hline 2’ means that the major and the next major sorted columns
will be used to separate identical rows regarding those two
columns.
• ‘:hline 3’, ‘:hline 4’, ... may be specified, but they may result
in too much hlines.
File: orgtbl-aggregate.info, Node: Cells processing, Next: Wide variety of inputs, Prev: hlines in the output table, Up: Top
11 Cells processing
*******************
*Calc* is the standard Emacs desktop calculator. Actual mathematical
computations are handled through Calc. This offers a lot of
flexibility.
* Menu:
* Where Calc interpretation happens?::
* Dates::
* Durations::
* Empty and malformed input cells::
* Symbolic computation::
* Intervals::
* Error or precision forms::
File: orgtbl-aggregate.info, Node: Where Calc interpretation happens?, Next: Dates, Up: Cells processing
11.1 Where Calc interpretation happens?
=======================================
Example of input table. Besides numbers, there are cells with
mathematical expressions like ‘20*30’, or just labels as ‘Red&Green’
without any mathematical meaning.
#+name: to_Calc_or_not_to_Calc
| Day | Color | Level |
|-----------+------------+--------|
| Monday | Red | 20*30 |
| Monday | Blue | 55+45 |
| Tuesday | Red | 1 |
| Tuesday | Red&Green | 2 |
| Tuesday | Blue+Green | 3 |
| Wednesday | Red | (27) |
| Wednesday | Red | (12+1) |
| Wednesday | Green | [15] |
Basically, Calc operates twice. For example in the formula
‘vsum(Level)’:
• Calc computes ‘Level’ for every input cell in the ‘Level’ column,
• then Calc computes ‘vsum()’ applied to the resulting list.
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vsum(Level)"
| Day | vsum(Level) |
|-----------+-------------|
| Monday | 700 |
| Tuesday | 6 |
| Wednesday | 55 |
#+END:
There are a few occasions were Calc computation does not happen:
‘vcount()’ and ‘vlist(X)’.
The ‘vcount()’ sub-formula is evaluated as the number of input rows
in each group, without Calc intervention. However, later on Calc can
handle this number in a formula as this one: ‘vsum(Level)/vcount()’
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vcount() vsum(Level)/vcount()"
| Day | vcount() | vsum(Level)/vcount() |
|-----------+----------+----------------------|
| Monday | 2 | 350 |
| Tuesday | 3 | 2 |
| Wednesday | 3 | 18.333333 |
#+END:
And of course when input cells do not have a mathematical meaning,
the result may be non-sens:
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vsum(Color)"
| Day | vsum(Color) |
|-----------+------------------------------------------------|
| Monday | Red + Blue |
| Tuesday | Red + error(3, '"Syntax error") + Blue + Green |
| Wednesday | 2 Red + Green |
#+END:
But it can also make sens. The last row, which aggregate ‘Wednesday’
rows, is computed as ‘2 Red + Green’. This is correct. This symbolic
result (as opposed to numerical result) shows the power of Calc as a
symbolic calculator.
The ‘vlist(X)’ formula is not handled by Calc at all. This formula
must appear alone (not embedded as part of a bigger formula). The cells
‘X’ are not interpreted by Calc. As a result, ‘vlist(X)’ produces a
cell which concatenates input cells verbatim. For instance, the input
cell ‘20*30’ is left as-is.
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day vlist(Color) vlist(Level)"
| Day | vlist(Color) | vlist(Level) |
|-----------+----------------------------+--------------------|
| Monday | Red, Blue | 20*30, 55+45 |
| Tuesday | Red, Red&Green, Blue+Green | 1, 2, 3 |
| Wednesday | Red, Red, Green | (27), (12+1), [15] |
#+END:
As a contrast, the formula ‘(Level)’ yields a list processed through
Calc. For instance, the ‘20*30’ formula is replaced by ‘600’.
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day (Color) (Level)"
| Day | (Color) | (Level) |
|-----------+------------------------------------------------+----------------|
| Monday | [Red, Blue] | [600, 100] |
| Tuesday | [Red, error(3, '"Syntax error"), Blue + Green] | [1, 2, 3] |
| Wednesday | [Red, Red, Green] | [27, 13, [15]] |
#+END:
Here we used parenthesis in ‘(Color)’ and ‘(Level)’ because otherwise
they would have been _key columns_. Instead of parenthesis, we can
embed such expressions in formulas, like ‘Level+1’:
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day Level+1"
| Day | Level+1 |
|-----------+----------------|
| Monday | [601, 101] |
| Tuesday | [2, 3, 4] |
| Wednesday | [28, 14, [16]] |
#+END:
To summarize, a column name embedded in a formula is evaluated as the
list of input cells, processed by Calc. Except for the ‘vlist(Column)’
formula where input cells are kept verbatim.
By the way, what is the meaning of the expression ‘Level*Level’? For
‘Monday’, it is ‘[600,100]*[600,100]’. Then Calc simplifies that as a
_vector product_: sum of individual products. ‘600^2+100^2’
#+BEGIN: aggregate :table "to_Calc_or_not_to_Calc" :cols "Day Level*Level Level+Level"
| Day | Level*Level | Level+Level |
|-----------+-------------+----------------|
| Monday | 370000 | [1200, 200] |
| Tuesday | 14 | [2, 4, 6] |
| Wednesday | 1123 | [54, 26, [30]] |
#+END:
File: orgtbl-aggregate.info, Node: Dates, Next: Durations, Prev: Where Calc interpretation happens?, Up: Cells processing
11.2 Dates
==========
Some aggregations are possible on dates. Example. Here is a source
table containing dates:
#+tblname: datetable
| Date |
|------------------------|
| [2035-12-22 Sat 09:01] |
| [2034-11-24 Fri 13:04] |
| [2030-09-24 Tue 13:54] |
| [2027-09-25 Sat 03:54] |
| [2023-02-26 Sun 16:11] |
| [2020-03-17 Tue 03:51] |
| [2018-08-21 Tue 00:00] |
| [2012-12-25 Tue 00:00] |
Here are the earliest and the latest dates, along with the average of
all input dates:
#+BEGIN: aggregate :table datetable :cols "vmin(Date) vmax(Date) vmean(Date)"
| vmin(Date) | vmax(Date) | vmean(Date) |
|------------------------+------------------------+-------------|
| <2012-12-25 Tue 00:00> | <2035-12-22 Sat 09:01> | 739448.44 |
#+END:
The average of all dates is a number? Actually, it is a date
expressed as the number of days since ‘[0000-12-31 Sun 00:00]’. To
force a number of days to be interpreted as a date, use the ‘date()’
function:
#+BEGIN: aggregate :table datetable :cols "date(vmean(Date))"
| date(vmean(Date)) |
|------------------------|
| <2025-07-16 Wed 10:29> |
#+END:
With the ‘date()’ function in mind, all kinds of dates handling can
be done. Example: the average of earliest and the latest dates is
different from the average of all dates:
#+BEGIN: aggregate :table datetable :cols "date(vmean(vmin(Date),vmax(Date))) date(vmean(Date))"
| date(vmean(vmin(Date),vmax(Date))) | date(vmean(Date)) |
|------------------------------------+------------------------|
| <2024-06-23 Sun 16:30> | <2025-07-16 Wed 10:29> |
#+END:
Note that ‘date()’ is not special to OrgAggregate. It can be used in
Org Mode spreadsheet formulas.
File: orgtbl-aggregate.info, Node: Durations, Next: Empty and malformed input cells, Prev: Dates, Up: Cells processing
11.3 Durations
==============
In Org Mode spreadsheet, durations have the forms ‘HH:MM’ or ‘HH:MM:SS’.
In OrgAggregate, when an input cell have one of those two forms, it is
converted into a number of seconds. For instance, ‘01:00’ is converted
into ‘3600’ and ‘00:00:07’ is converted into ‘7’.
There may be a single digit for hours, as in ‘7:12’ or more than two
as in ‘1255:45:00’.
To output such a form, use a formatter: ‘;T’; ‘;t’, ‘;U’. For
example, we have 3 durations as input, and we want the average of them:
#+name: some_durations
| dur |
|----------|
| 07:45:30 |
| 13:55 |
| 17:12 |
#+BEGIN: aggregate :table "some_durations" :cols "vmean(dur) vmean(dur);T vmean(dur);t vmean(dur);U"
| vmean(dur) | vmean(dur) | vmean(dur) | vmean(dur) |
|------------+------------+------------+------------|
| 46650 | 12:57:30 | 12.96 | 12:57 |
#+END:
• With no formatter, we get a number of seconds
• The ‘T’ formatter outputs the result as ‘HH:MM:SS’
• The ‘U’ formatter outputs the result as ‘HH:MM’
• The ‘t’ formatter converts the result into a number of hours (it
divides the number of seconds by 3600, and displays only two digits
after dot)
The Calc syntax for durations is also recognized:
HH@ MM'
HH@ MM' SS"
Example:
#+name: calc_durations
| dur |
|------------|
| 07@ 45' 30 |
| 13@ 55' |
| 17@ 12' |
#+BEGIN: aggregate :table "calc_durations" :cols "vmean(dur)"
| vmean(dur) |
|--------------|
| 12@ 57' 30." |
#+END:
File: orgtbl-aggregate.info, Node: Empty and malformed input cells, Next: Symbolic computation, Prev: Durations, Up: Cells processing
11.4 Empty and malformed input cells
====================================
The input table may contain malformed mathematical text. For instance,
a cell containing ‘5+’ is malformed, because an expression is missing
after the ‘+’ symbol. In this case, the value will be replaced by
‘error(2, '"Expected a number")’ which will appear in the aggregated
table, signaling the problem.
An input cell may be empty. In this case, it may be ignored or
converted to zero, depending on modifier flags ‘E’ and ‘N’.
The empty cells treatment
• makes no difference for ‘vsum’ and ‘count’.
• may result in zero for ‘prod’,
• change ‘vmean’ result,
• change ‘vmin’ and ‘vmax’, a possibly empty list of values resulting
in ‘inf’ or ‘-inf’
Some aggregation functions operate on two columns. If the two
columns have empty values at different locations, then they should be
interpreted as zero with the ‘NE’ modifier, otherwise the result will be
inconsistent.
Sometimes an input table may be malformed, with incomplete rows, like
this one:
| Color | Level | Quantity | Day |
|-------+-------+----------+-----------|
| Red | 30 | 11 | Monday |
| Blue | 25 | 3 | Monday |
|
| Blue | 33 | 18 | Tuesday |
| Red | 27 |
| Blue | 12 | 16 | Wednesday |
| Blue | 15 | 15 |
|
Missing cells are handled as though they were empty.
File: orgtbl-aggregate.info, Node: Symbolic computation, Next: Intervals, Prev: Empty and malformed input cells, Up: Cells processing
11.5 Symbolic computation
=========================
The computations are based on Calc, which is a symbolic calculator.
Thus, symbolic computations are built-in. Example:
This is the source table:
#+NAME: symtable
| Day | Color | Level | Quantity |
|-----------+-------+--------+----------|
| Monday | Red | 30+x | 11+a |
| Monday | Blue | 25+3*x | 3 |
| Tuesday | Red | 51+2*x | 12 |
| Tuesday | Red | 45-x | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12+x | 16 |
| Wednesday | Blue | 15 | 15-6*a |
| Thursday | Red | 39 | 24-5*a |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49+x | 30+9*a |
| Friday | Blue | 7 | 5+a |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
And here is the aggregated, symbolic result:
#+BEGIN: aggregate :table "symtable" :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+-----------------------+----------------|
| Monday | 2. x + 27.5 | a + 14 |
| Tuesday | 0.333333333334 x + 43 | 45 |
| Wednesday | x / 3 + 18 | 54 - 6 a |
| Thursday | x / 3 + 43. | 4 a + 83 |
| Friday | 8 | a + 22 |
#+END
Symbolic calculations are correctly performed on ‘x’ and ‘a’, which
are symbolic (as opposed to numeric) expressions.
Note that if there are empty cells in the input, they will be changed
to ‘nan’ _not a number_, and the whole aggregation will yield ‘nan’.
This is probably not the expected result.
The ‘N’ modifier (see *note Org Mode compatible formatters::) won’t
help, because even though it will replace empty cells with zero, it will
do the same for anything which does not look like a number. The best is
to just avoid empty cells when dealing with symbolic calculations.
File: orgtbl-aggregate.info, Node: Intervals, Next: Error or precision forms, Prev: Symbolic computation, Up: Cells processing
11.6 Intervals
==============
Org Mode tables and OrgAggregate backend engine being Emacs Calc,
intervals are seamlessly handled. An interval is made of two numerical
values separated by two dots.
Example of a spreadsheet. The third column is computed by
multiplying the first two:
#+name: int
| 3..5 | 10 | (30 .. 50) |
| 3..5 | -1..2 | (-5 .. 10) |
| 5 | 2 | 10 |
#+TBLFM: $3=$1*$2
Example of an aggregation. The sum of the third column is computed,
resulting in an interval:
#+BEGIN: aggregate :table "int" :cols "vsum($3)"
| vsum($3) |
|------------|
| (35 .. 70) |
#+END:
File: orgtbl-aggregate.info, Node: Error or precision forms, Prev: Intervals, Up: Cells processing
11.7 Error or precision forms
=============================
In Emacs Calc, an error form is a numerical value along with its
precision, both values separated by ‘+/-’.
The separation may also be the Unicode character ‘±’. In this
spreadsheet example, the second column is 10 times the first:
#+name: cert
| 12±2 | 120 +/- 20 |
| 55±3 | 550 +/- 30 |
| 9±0 | 90 |
| 15±1 | 150 +/- 10 |
| 21±1 | 210 +/- 10 |
#+TBLFM: $2=10*$1
Now, in the following example, OrgAggregate computes the sum of the
second column:
#+BEGIN: aggregate :table "cert" :cols "vsum($2);f2"
| vsum($2) |
|----------------|
| 1120 +/- 38.73 |
#+END:
The computation made by Emacs Calc assumes that all input values
follow a Gaussian distribution, and are independent variables. Then it
applies the textbook formula for combining Gaussian distributions.
Beware that your input values may not be independent with each other.
In this case, the resulting error will be slightly off (too small).
File: orgtbl-aggregate.info, Node: Wide variety of inputs, Next: Post-processing, Prev: Cells processing, Up: Top
12 Wide variety of inputs
*************************
As in any other Org Mode source block, the input table may come from
several places. OrgAggregate adds even more kinds of input.
Here we discus the ‘:table’ parameter.
* Menu:
* Standard Org Mode input::
* Virtual input table from Babel::
* An Org ID::
* CSV input::
* JSON input::
* Input slicing::
* Thecond filter: The cond filter.
* Virtual input columns::
File: orgtbl-aggregate.info, Node: Standard Org Mode input, Next: Virtual input table from Babel, Up: Wide variety of inputs
12.1 Standard Org Mode input
============================
The parameter after ‘:table’ may be:
• ‘mytable’: an ordinary Org Mode table in the same buffer, named
‘mytable’.
• ‘/some/dir/file.org:mytable’: an ordinary Org Mode table named
‘mytable’, in a distant Org file named ‘/some/dir/file.org’.
File: orgtbl-aggregate.info, Node: Virtual input table from Babel, Next: An Org ID, Prev: Standard Org Mode input, Up: Wide variety of inputs
12.2 Virtual input table from Babel
===================================
• ‘mybabel’: an Org Mode Babel block named ‘mybabel’ in the current
buffer, generating a table as its output, written in any language.
• ‘mybabel(param1=123,param2=456)’: passing parameters to an Org Mode
Babel block named ‘mybabel’ in the current buffer, generating a
table as its output, written in any language.
• ‘/some/dir/file.org:mybabel(param1=123,param2=456)’: an Org Mode
Babel block named ‘mybabel’ in a distant org file named
‘/some/dir/file.org’, called with parameters.
The input table may be the result of executing a Babel script. In
this case, the table is virtual in the sense that is appears nowhere.
(Babel is the Org Mode infrastructure to run scripts in any language,
like Python, R, C++, Java, D, Rust, shell, whatever, with inputs and
outputs connected to Org Mode).
Example:
Here is a script in Emacs Lisp which creates an Org Mode table.
#+name: ascript
#+begin_src elisp :colnames yes
`(
("label" value) ; cells are symbols or strings
hline
,@(cl-loop
for i from 10 to 20
collect
(list
(format "lbl-%c" (+ ?A (% i 3))) ; cell is a string
i))) ; cell is a number
#+end_src
If executed, the script would output this table:
#+RESULTS: ascript
| label | value |
|-------+-------|
| lbl-B | 10 |
| lbl-C | 11 |
| lbl-A | 12 |
| lbl-B | 13 |
| lbl-C | 14 |
| lbl-A | 15 |
| lbl-B | 16 |
| lbl-C | 17 |
| lbl-A | 18 |
| lbl-B | 19 |
| lbl-C | 20 |
But instead, OrgAggregate will execute the script and consume its
output:
#+BEGIN: aggregate :table "ascript" :cols "label vsum(value)"
| label | vsum(value) |
|-------+-------------|
| lbl-B | 58 |
| lbl-C | 62 |
| lbl-A | 45 |
#+END:
Here the parameter ‘:table’ specifies the name of the script to be
executed.
*Beware* of the ‘:results code’ parameter. It does not work as an
input for OrgAggregate. This is because in this case the Babel script
returns a string, not a table. Example:
'((a b c)
hline
(1 2 3))
Use ‘:results table’ or nothing instead.
File: orgtbl-aggregate.info, Node: An Org ID, Next: CSV input, Prev: Virtual input table from Babel, Up: Wide variety of inputs
12.3 An Org ID
==============
• ‘34cbc63a-c664-471e-a620-d654b26ffa31’: an identifier of an Org
Mode sub-tree. The sub-tree is supposed to contain an Org table
(which is retrieved) or a Babel script (which is executed).
Those Org Mode identifiers span all known Org Mode files. (Therefore
there is no need to specify a file). To add such an identifier, put the
cursor on the heading of the sub-tree, and type ‘M-x org-id-get-create’.
File: orgtbl-aggregate.info, Node: CSV input, Next: JSON input, Prev: An Org ID, Up: Wide variety of inputs
12.4 CSV input
==============
CVS input is specific to OrgAggregate. (Native Org Mode does not offers
those formats).
• ‘/some/dir/file.csv:(csv params…)’: a comma-separated-values file
in the CSV format, in the file ‘/some/dir/file.csv’.
• ‘name(csv params…)’: CSV formatted data within an Org block named
‘"name"’, in the current file.
• ‘/some/dir/file.org:name(csv params…)’: CSV formatted data within
an Org block named ‘"name"’, in a distant Org file.
The cells separators in the CSV files may be TAB, comma, or
semicolon, they are guessed and different separators may be mixed.
Any empty row in the CSV file is interpreted as an horizontal
separator (‘hline’ in Org table parlance).
Parameters may be:
• ‘header’: the first row in the CSV file is interpreted as a header
containing the column names.
• ‘colnames (column1 column2 column3 …)’: the column names are given
explicitly, in case the CSV file contains only data, no header.
In any case, the columns may be references as ‘$1, $2, $3, …’ as
usual.
When data is in an Org Mode file, it is supposed to be within a named
block of any kind. Example with a "quote" block:
#+name: mycsvdata
#+begin_quote
label,quantity
yellow,27
red,-61
yellow,41
red,-29
red,-17
#+end_quote
File: orgtbl-aggregate.info, Node: JSON input, Next: Input slicing, Prev: CSV input, Up: Wide variety of inputs
12.5 JSON input
===============
• ‘/some/dir/file.json:(json params…)’: a file containing a JSON
formatted table, in the file ‘/some/dir/file.csv’.
• ‘name(json params…)’: JSON formatted data within an Org block named
‘"name"’, in the current file.
• ‘/some/dir/file.org:name(json params…)’: JSON formatted data within
an Org block named ‘"name"’, in a distant Org file.
The accepted formats are a vector of vectors, or a vector of
hash-objects.
Currently, no ‘params’ are recognized.
Example of a vector of vectors:
[
["mon",12,"red" ],
["thu",34,"blue" ],
["wed",27,"green"],
["wed",21,"red" ],
["mon", 7,"blue" ],
…
]
Example of a vector of hash-objects:
[
{"day":"mon", "quty":12, "color":"red" },
{"day":"thu", "quty":34, "color":"blue" },
{"quty":27, "day":"wed", "color":"green"},
{"quty":21, "color":"red", "day":"wed" },
{"day":"mon", "quty": 7, "color":"blue" },
…
]
In the case of hash-objects, the keys are collected as the header of
the resulting table, in the order seen in the JSON file. In each
hash-object, the order of key-value pairs is irrelevant.
A header may be specified in the JSON file as a first vector,
followed by an hline (horizontal line). Example:
[
["day","quty","color"],
"hline",
["mon",12,"red" ],
["thu",34,"blue" ],
…
]
Horizontal lines may be ‘"hline"’, ‘[]’, ‘{}’, or ‘null’.
It is possible to mix both formats: vectors and hash-objects.
Example:
[
["quty","color"], // incomplete header
null, // horizontal line
{"day":"mon", "quty":12, "color":"red" }, // an hash-object
["thu", 34, "blue" ], // a vector
…
]
When data is in an Org Mode file, it is supposed to be within a named
block of any kind. Example with a "src" block for JavaScript:
#+name: myjsondata
#+begin_src js
[
["day","quty","color"],
"hline",
["mon",12,"red" ],
["thu",34,"blue" ],
…
]
#+end_src
File: orgtbl-aggregate.info, Node: Input slicing, Next: The cond filter, Prev: JSON input, Up: Wide variety of inputs
12.6 Input slicing
==================
Org Mode also provides for table slicing. All of the previous
references may be followed by an optional slicing. Examples:
• ‘mytable[0:5]’: retain only the first 6 rows of the input table; if
the table has a header, then it counts as 2 rows (the header and
the separation line). In this example, it would retain rows 0 and
1 for the header, and rows 2,3,4,5 for the content.
• ‘mytable[*,0:1]’: retain only the first 2 columns.
• ‘mytable[0:5,0:1]’: retain only the first 6 rows and the first 2
columns.
File: orgtbl-aggregate.info, Node: The cond filter, Next: Virtual input columns, Prev: Input slicing, Up: Wide variety of inputs
12.7 The :cond filter
=====================
This parameter is optional. If present, it specifies a Lisp expression
which tells whether or not a row should be kept. When the expression
evaluates to nil, the row is discarded.
Examples of useful expressions includes:
• ‘:cond (equal Color "Red")’
• to keep only rows where ‘Color’ is ‘Red’
• ‘:cond (> (string-to-number Quantity) 19)’
• to keep only rows for which ‘Quantity’ is more than ‘19’
• note the call to ‘string-to-number’; without this call,
‘Quantity’ would be used as a string
• ‘:cond (> (* (string-to-number Level) 2.5) (string-to-number
Quantity))’
• to keep only rows for which ‘2.5*Level > Quantity’
Beware with this example: ‘:cond (equal Color "Red")’. The input
table should not have a column named ‘Red’, otherwise the condition will
mean: _keep only rows with the same value in columns Color and Red_
As a special case, when ‘:cols’ parameter is not given, the result is
the same as ‘:cols "COL1 COL2 COL3..."’. All columns in the input table
are specified as key columns, and output in the resulting table.
This is useful when just filtering. But be aware that aggregation
still occurs. So duplicate input rows appear only once in the result.
File: orgtbl-aggregate.info, Node: Virtual input columns, Prev: The cond filter, Up: Wide variety of inputs
12.8 Virtual input columns
==========================
What if we want to aggregate on months? But the input table contains
only plain dates. Example:
#+name: without-months
| Date | Quantity |
|------------------+----------|
| [2037-03-12 thu] | 56.93 |
| [2037-03-25 wed] | 20.99 |
| [2037-04-07 tue] | 80.81 |
| [2037-04-20 mon] | 22.26 |
| [2037-05-03 sun] | 69.75 |
| [2037-05-16 sat] | 39.91 |
| [2037-05-29 fri] | 93.13 |
| [2037-06-11 thu] | 17.11 |
| [2037-06-24 wed] | 24.21 |
| [2037-07-07 tue] | 82.38 |
| [2037-07-20 mon] | 39.94 |
| [2037-08-02 sun] | 81.90 |
| [2037-08-15 sat] | 71.67 |
| [2037-08-28 fri] | 82.81 |
| [2037-09-10 thu] | 42.50 |
| [2037-09-23 wed] | 62.52 |
| [2037-10-06 tue] | 5.13 |
We would like to specify the aggregation over ‘month(Date)’. But
only plain columns may be used as aggregating keys.
One way is to add a ‘Month’ column to the input table. The modified
table looks like:
#+name: with-months
| Date | Quantity | Month |
|------------------+----------+-------|
| [2037-03-12 thu] | 56.93 | 3 |
| [2037-03-25 wed] | 20.99 | 3 |
| [2037-04-07 tue] | 80.81 | 4 |
| [2037-04-20 mon] | 22.26 | 4 |
| [2037-05-03 sun] | 69.75 | 5 |
| [2037-05-16 sat] | 39.91 | 5 |
| [2037-05-29 fri] | 93.13 | 5 |
| [2037-06-11 thu] | 17.11 | 6 |
| [2037-06-24 wed] | 24.21 | 6 |
| [2037-07-07 tue] | 82.38 | 7 |
| [2037-07-20 mon] | 39.94 | 7 |
| [2037-08-02 sun] | 81.90 | 8 |
| [2037-08-15 sat] | 71.67 | 8 |
| [2037-08-28 fri] | 82.81 | 8 |
| [2037-09-10 thu] | 42.50 | 9 |
| [2037-09-23 wed] | 62.52 | 9 |
| [2037-10-06 tue] | 5.13 | 10 |
#+TBLFM: $3=month($1)
OrgAggregate allows adding input columns like this computed ‘Month’
column, without modifying the input table. The ‘:precompute’ parameter
does that. Example:
#+BEGIN: aggregate :table "without-months" :cols "Month vsum(Quantity)" :precompute ("month(Date);'Month'")
| Month | vsum(Quantity) |
|-------+----------------|
| 3 | 77.92 |
| 4 | 103.07 |
| 5 | 202.79 |
| 6 | 41.32 |
| 7 | 122.32 |
| 8 | 236.38 |
| 9 | 105.02 |
| 10 | 5.13 |
#+END:
The specification ‘month(Date);'Month'’ means:
• add a third column to the input table,
• fill it with the formula ‘month(Date)’, which is a Calc formula,
• give this new column the ‘Month’ title,
• make this new column available for aggregation, as any other
column.
All this process is virtual. The input table is not modified in any
way.
If the title ‘Month’ is not specified, then the new virtual column
will be referred to as ‘$3’.
Note that here the title was specified with single quotes. This is
required to disambiguate with the format. The syntax is consistent with
the one used in the ‘:cols’ parameter and the one used by Org Mode
spreadsheet formulas.
The pre-computations may also be Lisp expressions, exactly like in
the usual Org table spreadsheet. In this example, we want to aggregate
on coarse bins. Bins are just hundredths of the first column:
#+name: want-bins
| 109 | 41.24 |
| 140 | 40.60 |
| 174 | 7.68 |
| 288 | 33.96 |
| 334 | 21.42 |
| 418 | 74.73 |
| 455 | 79.62 |
| 479 | 22.23 |
| 554 | 28.03 |
| 678 | 64.12 |
| 797 | 70.91 |
| 947 | 93.48 |
#+BEGIN: aggregate :table "want-bins" :cols "$3 vmean($2)" :precompute ("'(floor (/ (string-to-number $1) 100))")
| $3 | vmean($2) |
|----+-----------|
| 1 | 29.84 |
| 2 | 33.96 |
| 3 | 21.42 |
| 4 | 58.86 |
| 5 | 28.03 |
| 6 | 64.12 |
| 7 | 70.91 |
| 9 | 93.48 |
#+END:
Virtual columns may be formatted as any other column, with the same
syntax as in ‘:cols’ or in the Org table spreadsheet. For example here
we give it 2 digits after dot:
#+BEGIN: aggregate :table "want-bins" :cols "$3" :precompute "floor($1/100);%.2f"
| $3 |
|------|
| 1.00 |
| 2.00 |
| 3.00 |
| 4.00 |
| 5.00 |
| 6.00 |
| 7.00 |
| 9.00 |
#+END:
Of course, those additional virtual input columns may be used for
other purposes than key columns. They may enter in aggregating
formulas. Or they may be used by the optional row filter (the ‘:cond’
parameter). There is no difference between actual and virtual columns.
The ‘:precompute’ parameter may be:
• a list of strings, example:
("month(Date);'Month'" "day(Date);'Day'")
• a single string with fields separated by ‘::’, like in the
‘#+tblfm:’ tags of a spreadsheet. Example:
"month(Date);'Month' :: day(Date);'Day'"
• a string containing a single formula (actually this is a special
case of the previous one). Example:
"month(Date);'Month'"
File: orgtbl-aggregate.info, Node: Post-processing, Next: Pull & Push, Prev: Wide variety of inputs, Up: Top
13 Post-processing
******************
After OrgAggregate has generated the output table, it can be further
processed:
• additional columns may be added with the standard Org Mode
spreadsheet formulas.
• any algorithm in an exotic language (Python, R, C++, Emacs Lisp,
and so on) can be applied to the output.
* Menu:
* Spreadsheet formulas::
* Algorithm post processing::
* Grand total::
* Chaining::
File: orgtbl-aggregate.info, Node: Spreadsheet formulas, Next: Algorithm post processing, Up: Post-processing
13.1 Spreadsheet formulas
=========================
Additional columns can be specified for the resulting table. With a
previous example, adding a ‘:formula’ parameter, we specify a new column
‘$4’ which uses the aggregated columns. It is translated into a usual
‘#+TBLFM:’ spreadsheet line.
#+BEGIN: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)" :formula "$4=$2*$3"
| Day | vmean(Level) | vsum(Quantity) | |
|-----------+--------------+----------------+------|
| Monday | 27.5 | 14 | 385. |
| Tuesday | 43 | 45 | 1935 |
| Wednesday | 18 | 54 | 972 |
| Thursday | 43 | 83 | 3569 |
| Friday | 8 | 22 | 176 |
#+TBLFM: $4=$2*$3
#+END:
Moreover, if a ‘#+TBLFM:’ was already there, it survives aggregation
re-computations.
This happens in _pull mode_ only.
File: orgtbl-aggregate.info, Node: Algorithm post processing, Next: Grand total, Prev: Spreadsheet formulas, Up: Post-processing
13.2 Algorithm post processing
==============================
The aggregated table can be post-processed with the ‘:post’ parameter.
It accepts a Lisp ‘lambda’, a Lisp function, or a Babel block in any
exotic language (R, Python, C++, Emacs Lisp and so on).
The process receives the aggregated table as parameter in the form of
a Lisp expression. It can process it in any way it wants, provided it
returns a valid Lisp table.
A Lisp table is a list of rows. Each row is either a list of cells,
or the special symbol ‘hline’.
In this example, a ‘lambda’ expression adds a ‘hline’ and a row for
_Sunday_.
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)" :post (lambda (table) (append table '(hline (Sunday "0.0"))))
| Day | vsum(Quantity) |
|-----------+----------------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
|-----------+----------------|
| Sunday | 0.0 |
#+END:
The ‘lambda’ can be moved to a ‘defun’. The function is then passed
to the ‘:post’ parameter:
#+begin_src elisp
(defun my-function (table)
(append table
'(hline (Sunday "0.0"))))
#+end_src
... :post my-function
The ‘:post’ parameter can also refer to a Babel Block. Example:
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)" :post "my-babel-block(tbl=*this*)"
...
#+END:
#+name: my-babel-block
#+begin_src elisp :var tbl=""
(append tbl
'(hline (Sunday "0.0")))
#+end_src
*Beware!* You may want to add ‘:colnames t’ to your Babel block.
Otherwise the table’s header will be lost.
File: orgtbl-aggregate.info, Node: Grand total, Next: Chaining, Prev: Algorithm post processing, Up: Post-processing
13.3 Grand total
================
She (the user) may be tempted to add the grand total at the bottom of
the aggregation. Example of such a temptation:
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
|-----------+--------------+----------------|
| Total | | 218 |
#+TBLFM: @7$3=vsum(@I..@II)
#+end
With OrgAggreagate post-processing, it is easy.
1. Just put in place a small algorithm to add two lines.
:post (append *this* '(hline ("Total" "" "")))
▲ ▲ ▲
╰─────╮ │ │
the aggregated table╶─╯ │ │
one horizontal line╶─────╯ │
an empty cell to recieve the total╶──────╯
The ‘*this*’ Lisp variable contains the aggregated table, just while
the post-processing takes place. The ‘append’ Lisp function adds two
rows to the aggregated table, and returns the amended table.
Note that the ‘:post’ parameter may be:
• a small Lisp expression, as in this example,
• a lambda expression, which parameter is the aggregated table,
• the name of a Lisp function, which is passed the aggregate table,
• the name of a Babel block, written in any supported language.
• Fill the additional cell with a formula for the total.
@>$2=vsum(@I..@II)
▲ ▲ ▲ ▲ ▲
│ │ │ │ ╰──────────────╮
│ │ │ ╰────────────────╮ │
│ │ ╰─────────────────╮ │ │
│ ╰────────────╮ │ │ │
╰──────────╮ │ │ │ │
last line╶─╯ │ │ │ │
second column╶─╯ │ │ │
sum all values between╶─╯ │ │
the first horizontal line╶─╯ │
and the second one╶──────────╯
In this way, the grand-total will be recomputed each time the
aggregation is refreshed (‘C-c C-c’). Note the use of ‘@>$2’ for the
coordinates of the cell receiving the total, instead of, for instance
‘@7$3’. This ensures that the formula will continue to be applied on
the last row, even if the aggregated table grows later on. The same
idea applies for the ‘@I..@II’ range, instead of, for instance ‘@2..@6’.
Even though OrgAggregate offers the user a versatil post-processing
to add a grand total, there are many reasons not to. If she does, she
is quietly entering the same nightmare which plagues spreadsheets.
Everything will become harder and harder to maintain.
It seems natural to add a grand total right below the column. But
suppose that now she also want the standard deviation of this same
column. Where to put it? There is a blank cell just left of the grand
total. She puts the standard deviation there:
#+begin: aggregate :table original :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
|-----------+--------------+----------------|
| Total | 27.409852 | 218 |
#+TBLFM: @7$3=vsum(@I..@II)::@7$2=vsdev(@I$3..@II$3)
#+end
But then this ‘27.409852’ looks like the grand total of the second
column, but it is not.
Later on, she may want to further process this aggregated table, for
example to plot it. The grand total row will be an annoyance. It will
be tedious to get ride of it.
She should instead consider creating a second aggregation with the
grand total:
#+begin: aggregate :table original :cols "vsdev(Level) vsum(Quantity)"
| vsdev(Level) | vsum(Quantity) |
|--------------+----------------|
| 27.409852 | 218 |
#+end
Easy, maintainable, no awkward decisions to remember and document.
She keeps it simple.
File: orgtbl-aggregate.info, Node: Chaining, Prev: Grand total, Up: Post-processing
13.4 Chaining
=============
The result of an aggregation may become the source of further
processing. To do that, just add a ‘#+NAME:’ or ‘#+TBLNAME:’ line just
above the aggregated table. Here is an example of a double aggregation:
#+NAME: squantity
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)"
| Day | SQuantity |
|-----------+-----------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
#+TBLFM: @1$2=SQuantity
#+END:
#+BEGIN: aggregate :table "squantity" :cols "vsum(SQuantity)"
| vsum(SQuantity) |
|-----------------|
| 218 |
#+END:
Note the spreadsheet cell formula ‘@1$2=SQuantity’, which changes the
column heading from it default ‘vsum(Quantity)’ to ‘SQuantity’. This
new heading will survive any refresh.
Sometimes the name of the aggregated table is not found by some babel
block referencing it (Gnuplot blocks are among them). To fix that, just
exchange the ‘#+NAME:’ and ‘#+BEGIN:’ lines:
#+BEGIN: aggregate :table original :cols "Day vsum(Quantity)"
#+NAME: squantity
| Day | SQuantity |
|-----------+-----------|
| Monday | 14 |
| Tuesday | 45 |
| Wednesday | 54 |
| Thursday | 83 |
| Friday | 22 |
#+TBLFM: @1$2=SQuantity
#+END:
The ‘#.NAME:’ line will survive when recomputing the aggregation (as
‘#.TBLFM:’ line survives)
File: orgtbl-aggregate.info, Node: Pull & Push, Next: Debugging, Prev: Post-processing, Up: Top
14 Pull & Push
**************
Two modes are available: _pull_ & _push_.
* Menu:
* Pull mode::
* Push mode::
* Pull or push ?::
File: orgtbl-aggregate.info, Node: Pull mode, Next: Push mode, Up: Pull & Push
14.1 Pull mode
==============
In the _pull_ mode, we use so called _"dynamic blocks"_. The resulting
table knows how to build itself.
Example:
We have a source table which is unaware that it will be derived in an
aggregated table:
#+NAME: source1
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
We create somewhere else a _dynamic block_ which carries the
specification of the aggregation:
#+BEGIN: aggregate :table "source1" :cols "Day vmean(Level) vsum(Quantity)"
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+END
Typing ‘C-c C-c’ in the dynamic block recomputes it freshly.
File: orgtbl-aggregate.info, Node: Push mode, Next: Pull or push ?, Prev: Pull mode, Up: Pull & Push
14.2 Push mode
==============
In _push_ mode, the source table drives the creation of derived tables.
We specify the wanted results in ‘#+ORGTBL: SEND’ directives (as many as
desired):
#+ORGTBL: SEND derived1 orgtbl-to-aggregated-table :cols "vmean(Level) vsum(Quantity)"
#+ORGTBL: SEND derived2 orgtbl-to-aggregated-table :cols "Day vmean(Level) vsum(Quantity)"
| Day | Color | Level | Quantity |
|-----------+-------+-------+----------|
| Monday | Red | 30 | 11 |
| Monday | Blue | 25 | 3 |
| Tuesday | Red | 51 | 12 |
| Tuesday | Red | 45 | 15 |
| Tuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12 | 16 |
| Wednesday | Blue | 15 | 15 |
| Thursday | Red | 39 | 24 |
| Thursday | Red | 41 | 29 |
| Thursday | Red | 49 | 30 |
| Friday | Blue | 7 | 5 |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
We must create the receiving blocks somewhere else in the same file:
#+BEGIN RECEIVE ORGTBL derived1
#+END RECEIVE ORGTBL derived1
#+BEGIN RECEIVE ORGTBL derived2
#+END RECEIVE ORGTBL derived2
Then we come back to the source table and type ‘C-c C-c’ with the
cursor on the 1st pipe of the table, to refresh the derived tables:
#+BEGIN RECEIVE ORGTBL derived1
| vmean(Level) | vsum(Quantity) |
|---------------+----------------|
| 27.9285714286 | 218 |
#+END RECEIVE ORGTBL derived1
#+BEGIN RECEIVE ORGTBL derived2
| Day | vmean(Level) | vsum(Quantity) |
|-----------+--------------+----------------|
| Monday | 27.5 | 14 |
| Tuesday | 43 | 45 |
| Wednesday | 18 | 54 |
| Thursday | 43 | 83 |
| Friday | 8 | 22 |
#+END RECEIVE ORGTBL derived2
File: orgtbl-aggregate.info, Node: Pull or push ?, Prev: Push mode, Up: Pull & Push
14.3 Pull or push ?
===================
Pull & push modes use the same engine in the background. Thus, using
either is just a matter of convenience.
Pull mode is the most straightforward. Also the wizard operates on
the pull mode only. Almost all examples in this documentation are in
pull mode. If you cannot decide, just use the pull mode.
Glitch: in push mode you may see strange output like ‘\_{}’. This is
an escape generated by Org Mode (nothing to do with OrgAggregate). It
happens for the following characters: ‘&%#_^’ To disable that, in the
‘#+ORGTBL: SEND’ line, add this parameter: ‘:no-escape true’
File: orgtbl-aggregate.info, Node: Debugging, Next: Tricks, Prev: Pull & Push, Up: Top
15 Debugging
************
The work of OrgAggregate is to hand out pieces of the input table to
Calc (the Emacs calculator).
Is some intricate cases, it may be useful to see what is going on.
The debugging formatters come handy.
* Menu:
* Seeing the $ forms::
* Seeing Calc formulas before evaluation::
* Seeing Lisp internal form of Calc formulas::
* Example of debugging vsum(nn^2)::
* Summary of debugging formatters::
File: orgtbl-aggregate.info, Node: Seeing the $ forms, Next: Seeing Calc formulas before evaluation, Up: Debugging
15.1 Seeing the $ forms
=======================
Here is an example input table:
#+name: inputdebug
| nn | aa |
|------+----|
| 1.23 | a |
| 7.65 | b |
| 8.46 | c |
|------+----|
| 2.44 | d |
| 6.81 | e |
And here is an aggregation to debug:
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10) vsum(aa+7)"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-------------+----------------|
| 0 | 173.4 | a + b + c + 21 |
| 1 | 92.5 | d + e + 14 |
#+END:
So far so good. But we would like to know what Calc did. To do so
let us add the ‘c’ formatter.
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);c vsum(aa+7);c"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-------------+------------|
| 0 | vsum($1*10) | vsum($2+7) |
| 1 | vsum($1*10) | vsum($2+7) |
#+END:
Each output cell now contains the formula, with column names replaced
by dollar equivalent forms. But without any further processing.
File: orgtbl-aggregate.info, Node: Seeing Calc formulas before evaluation, Next: Seeing Lisp internal form of Calc formulas, Prev: Seeing the $ forms, Up: Debugging
15.2 Seeing Calc formulas before evaluation
===========================================
Let us go one step further with the ‘C’ formatter:
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);C vsum(aa+7);C"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+-----------------------------+---------------------|
| 0 | vsum([1.23, 7.65, 8.46] 10) | vsum([a, b, c] + 7) |
| 1 | vsum([2.44, 6.81] 10) | vsum([d, e] + 7) |
#+END:
The dollar forms were replaced by Calc vectors made of input cells.
No foldings or simplifications went on. The vectors are slices of
columns, selected by OrgAggregate in response of the ‘hline’
aggregation.
We see that multiplying by ‘10’ or adding ‘7’ is done on a Calc
vector. It happens that Calc knows how to multiply or add something to
a vector. OrgAggregate does not perform those operations, it delegates
them to Calc.
File: orgtbl-aggregate.info, Node: Seeing Lisp internal form of Calc formulas, Next: Example of debugging vsum(nn^2), Prev: Seeing Calc formulas before evaluation, Up: Debugging
15.3 Seeing Lisp internal form of Calc formulas
===============================================
We can also view the same results, formatted as Lisp forms (rather than
Calc forms) with the ‘Q’ formatter:
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn*10);Q vsum(aa+7);Q"
| hline | vsum(nn*10) | vsum(aa+7) |
|-------+---------------------------------------------------------------------------+-----------------------------------------------------------------------|
| 0 | (calcFunc-vsum (* (vec (float 123 -2) (float 765 -2) (float 846 -2)) 10)) | (calcFunc-vsum (+ (vec (var a var-a) (var b var-b) (var c var-c)) 7)) |
| 1 | (calcFunc-vsum (* (vec (float 244 -2) (float 681 -2)) 10)) | (calcFunc-vsum (+ (vec (var d var-d) (var e var-e)) 7)) |
#+END:
This is the internal, Lisp representation of Calc formulas.
File: orgtbl-aggregate.info, Node: Example of debugging vsum(nn^2), Next: Summary of debugging formatters, Prev: Seeing Lisp internal form of Calc formulas, Up: Debugging
15.4 Example of debugging vsum(nn^2)
====================================
Beware of a formula like ‘vsum(nn^2)’. This gives the expected result,
although not in the obvious way. Let us see what happens, thanks to the
‘C’ debugging formatter:
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn^2);C"
| hline | vsum(nn^2) |
|-------+----------------------------|
| 0 | vsum([1.23, 7.65, 8.46]^2) |
| 1 | vsum([2.44, 6.81]^2) |
#+END:
We are not summing squares. We are squaring Calc vectors. Calc
being a mathematical tool, it interprets the product of two vectors as
the sum of the products element-wise, as a mathematician would do. Then
‘vsum’ is applied on a single resulting value. So ‘vsum’ is useless in
this case. That can be confirmed:
#+BEGIN: aggregate :table "inputdebug" :cols "hline vsum(nn^2) nn^2 vprod(nn^2)"
| hline | vsum(nn^2) | nn^2 | vprod(nn^2) |
|-------+------------+---------+-------------|
| 0 | 131.607 | 131.607 | 131.607 |
| 1 | 52.3297 | 52.3297 | 52.3297 |
#+END:
Therefore, changing ‘vsum’ to ‘vprod’ does not change the result.
This can be unexpected.
File: orgtbl-aggregate.info, Node: Summary of debugging formatters, Prev: Example of debugging vsum(nn^2), Up: Debugging
15.5 Summary of debugging formatters
====================================
To summarize the debugging settings:
• ‘c’: output Calc formula
• ‘C’: output Calc formula with dollar forms substituted by actual
input data
• ‘q’: output Lisp formula
• ‘Q’: output Lisp formula with column forms substituted by actual
input data
File: orgtbl-aggregate.info, Node: Tricks, Next: Installation, Prev: Debugging, Up: Top
16 Tricks
*********
This chapter collects some tricks that may be useful.
* Menu:
* Sorting: Sorting (1).
* A few lowest or highest values::
* Span of values::
* No aggregation::
File: orgtbl-aggregate.info, Node: Sorting (1), Next: A few lowest or highest values, Up: Tricks
16.1 Sorting
============
#+name: trick_table_1
| column |
|--------|
| 677 |
| 713 |
| 459 |
| 537 |
| 881 |
When several cells of a column need to be sorted, the Calc
‘calc-sort()’ function is handy:
#+BEGIN: aggregate :table "trick_table_1" :cols "(column) sort(column)"
| (column) | sort(column) |
|---------------------------+---------------------------|
| [677, 713, 459, 537, 881] | [459, 537, 677, 713, 881] |
#+END:
• ‘(column)’ gives the list of values to aggregate, without
aggregating them.
• ‘sort(column)’ gives the same list sorted in ascending order.
File: orgtbl-aggregate.info, Node: A few lowest or highest values, Next: Span of values, Prev: Sorting (1), Up: Tricks
16.2 A few lowest or highest values
===================================
Used with ‘subvec()’, ‘sort()’ can retrieve the two lowest or the two
highest values:
#+BEGIN: aggregate :table "trick_table_1" :cols "subvec(sort(column),1,3) subvec(sort(column),count()-1)"
| subvec(sort(column),1,3) | subvec(sort(column),count()-1) |
|--------------------------+--------------------------------|
| [459, 537] | [713, 881] |
#+END:
• ‘subvec(...,1,3)’ extracts the two first values: from ‘1’ to ‘3’
excluded.
• ‘subvec(...,count()-1)’ extracts the two last values, numbered
‘count()-1’ and ‘count()’
And of course we may retrieve the average of the two first and the
two last values:
#+BEGIN: aggregate :table "trick_table_1" :cols "vmean(subvec(sort(column),1,3)) vmean(subvec(sort(column),count()-1))"
| vmean(subvec(sort(column),1,3)) | vmean(subvec(sort(column),count()-1)) |
|---------------------------------+---------------------------------------|
| 498 | 797 |
#+END:
File: orgtbl-aggregate.info, Node: Span of values, Next: No aggregation, Prev: A few lowest or highest values, Up: Tricks
16.3 Span of values
===================
‘vmin()’ and ‘vmax()’ can compute the span of aggregated values:
#+BEGIN: aggregate :table "trick_table_1" :cols "vmin(column) vmax(column) vmax(column)-vmin(column)"
| vmin(column) | vmax(column) | vmax(column)-vmin(column) |
|--------------+--------------+---------------------------|
| 459 | 881 | 422 |
#+END:
File: orgtbl-aggregate.info, Node: No aggregation, Prev: Span of values, Up: Tricks
16.4 No aggregation
===================
Why would one want to use OrgAggregate while not aggregating? To
benefit from the other features of OrgAggregate:
• column rearrangement
• sorting
• formatting
• ‘#+TBLFM’ survival
• row filtering
• preprocess
• postprocess
To do so, mention the virtual column ‘@#’ in ‘:cols’ and make it
invisible with ‘;<>’. As ‘@#’ is different for each row, the
aggregation will consider each row as a separate group. Therefore, no
aggregation on another column will do anything more.
For example, here we:
• put ‘Color’ as the first column (it is the second in the input),
• ignore the ‘Day’ column,
• sort by ‘Level’,
• compute ‘Quantity/7’,
• format it with 2 digits after dot.
#+BEGIN: aggregate :table "original" :cols "@#;<> Color Level;^n vmax(Quantity/7);'Q10';f2"
| Color | Level | Q10 |
|-------+-------+------|
| Blue | 6 | 1.14 |
| Blue | 7 | 0.71 |
| Blue | 11 | 1.29 |
| Blue | 12 | 2.29 |
| Blue | 15 | 2.14 |
| Blue | 25 | 0.43 |
| Red | 27 | 3.29 |
| Red | 30 | 1.57 |
| Blue | 33 | 2.57 |
| Red | 39 | 3.43 |
| Red | 41 | 4.14 |
| Red | 45 | 2.14 |
| Red | 49 | 4.29 |
| Red | 51 | 1.71 |
#+END:
We used the ‘vmax()’ aggregating function on ‘Quantity/7’, because
otherwise we would get a vector with a single value. As there is a
single value, any aggregating function will do the trick: ‘vmin()’,
‘head()’, ‘rtail()’, ‘vsum()’, ‘vprod()’, ‘vmean()’, ‘vgmean()’,
‘vhmean()’, ‘vspan()’, ‘vmedian()’.
File: orgtbl-aggregate.info, Node: Installation, Next: Authors contributors, Prev: Tricks, Up: Top
17 Installation
***************
Emacs package on Melpa: add the following lines to your ‘.emacs’ file,
and reload it.
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/") t)
(package-initialize)
You may also customize this variable:
M-x customize-variable package-archives
Then browse the list of available packages and install
‘orgtbl-aggregate’
M-x package-list-packages
Alternatively, you can download the lisp file, and load it:
(load-file "orgtbl-aggregate.el")
File: orgtbl-aggregate.info, Node: Authors contributors, Next: Changes, Prev: Installation, Up: Top
18 Authors, contributors
************************
Authors
• Thierry Banel, tbanelwebmin at free dot fr, inception &
implementation.
• Michael Brand, Calc unleashed, ‘#+TBLFM’ survival, empty input
cells, formatters.
Contributors
• Eric Abrahamsen, non-ASCII column names
• Alejandro Erickson, quoting non alphanumeric column names
• Uwe Brauer, simpler example in documentation, take
org-calc-default-modes preferences into account
• Peking Duck, fixed obsolete letf function
• Bill Hunker, discovered ‘\_{}’ escape
• Dirk Schmitt, surviving ‘#.NAME:’ line
• Dale Sedivec, case insensitive ‘#+NAME:’ tags
• falloutphil, underscore in column names
• Baudilio Tejerina, t, T, U formatters
• Marco Pas, bug comparing empty string
• wuqui, sorting output table, filtering only
• Nicolas Viviani, output hlines
• Nils Lehmann, support old versions of the rx library
• Shankar Rao, ‘:post’ post-processing
• Misohena (), double width
Japanese characters (string-width vs. length)
• Kevin Brubeck Unhammer, ignore formatting cookies
• Tilmann Singer, more flexibility in duration format
• Piotr Panasiuk, ‘#+CAPTION:’ and any tags survive
• Luis Miguel Hernanz, fix regex bug
• Jason Hemann, output column names no longer have quotes
• Tilmann Singer, computed aggregating bins, ‘"month(Date)"’ in his
use case
File: orgtbl-aggregate.info, Node: Changes, Next: GPL 3 License, Prev: Authors contributors, Up: Top
19 Changes
**********
Top: earliest change. Bottom: latest change.
• Wizard now correctly asks for columns with ‘$1, $2...’ names when
table header is missing
• Handle tables beginning with hlines
• Handle non-ASCII column names
• ‘:formula’ parameter and ‘#+TBLFM’ survival
• Empty cells are ignored.
• Empty output upon too small input set
• Fix ordering of output values
• Aggregations formulas may now be arbitrary expressions
• Table headers (and the lack of) are better handled
• Modifiers and formatters can now be specified as in the spreadsheet
• Aggregation function names can optionally have a leading ‘v’, like
‘sum’ & ‘vsum’
• Increased performance on large data sets
• Tables can be named with ‘#+NAME:’ besides ‘#+TBLNAME:’
• Document Melpa installation
• Support quoting of column names, like "a.b" or ’c/d’
• Disable ‘\_{}’ escape
• ‘#+NAME:’ inside ‘#+BEGIN:’ survives
• Missing input cells handled as empty ones
• Back-port Org Mode ‘9.4’ speed up
• Increase performance when inserting result into the buffer
• Aligned output in push mode
• Added a hash-table to speedup aggregation
• Back-port org-table-to-lisp which is now much faster
• ‘vlist(X)’ now yields input cells verbatim were ‘(X)’ yields Calc
processed input cells
• Document dates handling and the ‘date()’ function
• Implement ‘HH:MM:SS’ durations and ‘T’, ‘t’, ‘U’ formatters
• Sort output
• Create hlines in the output
• Missing :cond parameter means all columns
• Remove ‘C-c C-x i’, use standard ‘C-c C-x x’ instead
• Avoid name collision between Calc functions and columns
• More readable & faster code
• Support for old versions of the rx library
• ‘:post’ post-processing
• Propagate multiple rows source header to the aggregated header
• Ignore data rows containing formatting cookies
• Follow Org Mode way of handling Calc settings in Lisp code
• Hours in durations are no longer restricted to 2 digits
• 3x speedup ‘org-table-to-lisp’ and avoid Emacs 27 to 30
incompatibilities
• ‘#+CAPTION:’ and any other tag survive inside ‘#+BEGIN:’
• Output column names are now stripped from quotes, better reflecting
input names.
• Table-of-contents in README.org (thanks org-make-toc)
• Add formatters ‘c’ ‘C’ ‘q’ ‘Q’ (useful for debugging or
understanding OrgAggregate)
• Formulas involving ‘hline’ like ‘vmean(hline*10)’ are now taken
into account
• Documentation is now integrated right into Emacs in the ‘info’
format. Type ‘M-: (info "orgtbl-aggregate")’
• Input table may now be the result of a Babel script (virtual
table).
• Better handling of user errors in the ‘:post’ directive.
• Speedup of resulting table recalculation when there are formulas in
‘#+tblfm:’ or in ‘:formula’. The overall aggregation may be up to
x6 faster and ÷5 less memory hungry.
• Circumvent an Org Mode bug in case there are a column-formula along
with a cell-formula, the cell-one not being calculated. (Bonus:
15% speedup).
• Fix issue #24: bug in date parsing.
• Virtual pre-computed input columns.
• Better explanation of the input table reference syntax, including
distant tables and virtual table produced by Babel blocks.
• Support for CSV and JSON formatted input tables.
• New ‘@#’ virtual column giving the number of each row, pretty much
like the Org table spreadsheet ‘@#’ virtual column.
• Header and column names can be specified for CSV input tables, as
well as horizontal separators (‘hline’).
• Aggregation of the titles of this README.
• New free-form wizard.
• Illustrate README with Uniline graphics.
• JSON and CSV input tables can now live inside Org Mode blocks.
• Example for computing a refreshable grand total.
• Document intervals and error-forms handling.
• Special columns ‘@#’ and ‘hline’ are handled by transpose.
File: orgtbl-aggregate.info, Node: GPL 3 License, Prev: Changes, Up: Top
20 GPL 3 License
****************
Copyright (C) 2013-2026 Thierry Banel
orgtbl-aggregate is free software: you can redistribute it and/or
modify it under the terms of the GNU General Public License as published
by the Free Software Foundation, either version 3 of the License, or (at
your option) any later version.
orgtbl-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
Tag Table:
Node: Top209
Node: New3098
Node: Examples3529
Node: A very simple example3799
Node: Demonstrate sum and average computing6154
Node: Example without days9492
Node: Example of counting each combination10360
Node: Stop reading here! 80/2013040
Node: Name your input table13520
Node: Create an aggregation block14173
Node: Refresh the aggregation15095
Node: Equivalent in SQL R Datamash el-tblfn Awk C++15701
Node: SQL equivalent16346
Node: R equivalent16692
Node: Datamash equivalent17226
Node: el-tblfn17800
Node: Awk equivalent18467
Node: C++ equivalent19197
Node: Wizards20321
Node: Guiding (traditional) wizard20617
Node: Experimental free form wizard22786
Node: The cols parameter24832
Node: Names of input columns25503
Node: Grouping specifications in cols26578
Node: The hline column27134
Node: The @# column29922
Node: Aggregation formulas in cols32403
Node: Correlation of two columns39944
Node: [Almost) any expression can be specified41683
Node: Column names43157
Node: Input table with or without a header43433
Node: Column names of the input table44826
Node: Multiple lines header45995
Node: Custom column names48912
Node: Formatters51322
Node: Org Mode compatible formatters51548
Node: Debugging formatters53749
Node: Discarding an output column54431
Node: Sorting57221
Node: Example with one sorting column57427
Node: Several sorting columns58708
Node: hlines in the output table61060
Node: Output hlines depends on sorting columns61458
Node: Example with hline 264354
Node: Cells processing69446
Node: Where Calc interpretation happens?69941
Node: Dates75418
Node: Durations77423
Node: Empty and malformed input cells79257
Node: Symbolic computation80927
Node: Intervals83224
Node: Error or precision forms84020
Node: Wide variety of inputs85191
Node: Standard Org Mode input85738
Node: Virtual input table from Babel86215
Node: An Org ID88816
Node: CSV input89417
Node: JSON input90938
Node: Input slicing93339
Node: The cond filter94062
Node: Virtual input columns95562
Node: Post-processing101035
Node: Spreadsheet formulas101576
Node: Algorithm post processing102674
Node: Grand total104637
Node: Chaining109459
Node: Pull & Push111148
Node: Pull mode111382
Node: Push mode113085
Node: Pull or push ?115293
Node: Debugging116027
Node: Seeing the $ forms116550
Node: Seeing Calc formulas before evaluation117760
Node: Seeing Lisp internal form of Calc formulas118894
Node: Example of debugging vsum(nn^2)120103
Node: Summary of debugging formatters121532
Node: Tricks122028
Node: Sorting (1)122306
Node: A few lowest or highest values123121
Node: Span of values124422
Node: No aggregation124981
Node: Installation126835
Node: Authors contributors127477
Node: Changes129105
Node: GPL 3 License133494
End Tag Table
Local Variables:
coding: utf-8
End:
================================================
FILE: tests/distant-tests.org
================================================
# -*- coding:utf-8; -*-
#+TITLE: Distant tables for testing Orgtbl Aggregate
Copyright (C) 2013-2026 Thierry Banel
org-aggregate is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
org-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
* Table by name
#+name: distanttable
| tag | val |
|-----+-------|
| A | 11.58 |
| BB | 98.43 |
| CCC | 87.74 |
| BB | 87.97 |
| A | 84.32 |
| A | 83.79 |
| CCC | 4.32 |
| A | 31.07 |
| BB | 70.82 |
| BB | 32.32 |
| A | 88.22 |
| CCC | 8.61 |
| BB | 55.17 |
| BB | 52.73 |
| A | 33.96 |
| CCC | 23.42 |
| CCC | 91.62 |
| BB | 56.19 |
| A | 10.78 |
| A | 61.71 |
* Babel by name and parameter
#+name: distantbabel
#+begin_src elisp :colnames yes :var factor=29
(append
(list
'(tag value inv)
'hline)
(cl-loop
for i from 1 to 50
collect
(list
(aref [A BB CCC] (% i 3))
(% (* i factor) 100)
(/ 1000.0 (+ 999.0 i)))))
#+end_src
* Table by ID
:PROPERTIES:
:ID: 55ab27a2-c44b-4a14-9ba4-f6879375207d
:END:
| ref | val |
|------+-------|
| S | 685.6 |
| TT | 229.1 |
| UUU | 945.2 |
| VVVV | 115.6 |
| VVVV | 859.1 |
| TT | 767.3 |
| UUU | 242.8 |
| S | 934.5 |
| S | 349.2 |
| UUU | 227.8 |
| VVVV | 784.8 |
| TT | 225.4 |
| VVVV | 880.6 |
| UUU | 228.0 |
| TT | 882.2 |
| TT | 157.4 |
| VVVV | 504.4 |
| S | 521.6 |
| TT | 90.9 |
* CSV in distant Org file
#+name: distantcsv
#+begin_quote
label,quantity
jes,23
ne,-65
jes,47
ne,-22
ne,-19
#+end_quote
* JSON in distant Org file
#+name: distantjson
#+begin_src js
[
[ 42, "univ"],
[123, "jes" ],
[321, "jes" ],
[111, "jes" ],
[-77, "ne" ],
[-99, "ne" ]
]
#+end_src
================================================
FILE: tests/geography-a.csv
================================================
Tokyo Japan 37131070 37173748
Delhi India 34598951 33741241
Shanghai China 30365228 29973570
Dhaka Bangladesh 24561693 23958442
Cairo Egypt 23095986 22623336
Sao Paulo Brazil 23045227 22830606
Mexico City Mexico 22831373 22471468
Beijing China 22559204 22155716
Mumbai India 22012722 21690477
Osaka Japan 18990205 19040445
Chongqing China 18100872 17809422
Karachi Pakistan 18059805 17637893
Kinshasa DR Congo 17815364 17025505
Lagos Nigeria 17124998 16554804
Istanbul Turkey 16211581 16054896
Kolkata India 15904385 15535352
Buenos Aires Argentina 15714124 15634092
Manila Philippines 15211511 14917612
Guangzhou China 14870254 14634025
Lahore Pakistan 14835678 14359466
Tianjin China 14729021 14417999
Bangalore India 14387359 14059333
Rio de Janeiro Brazil 13880630 13774171
Shenzhen China 13501772 13330796
Moscow Russia 12768223 12699759
Chennai India 12353002 12009954
Bogota Colombia 11779275 11660428
Jakarta Indonesia 11628728 11478423
Lima Peru 11529982 11388304
Bangkok Thailand 11415533 11195528
Paris France 11352823 11304387
Hyderabad India 11307135 11046182
Nanjing China 10208049 9935614
Luanda Angola 10049628 9648709
Seoul South Korea 10059272 9991484
Chengdu China 10001659 9809118
London United Kingdom 9818142 9723207
Ho Chi Minh City Vietnam 9798896 9541155
Tehran Iran 9738111 9606808
Nagoya Japan 9544065 9544935
Xi-an China 9253706 9000317
Ahmedabad India 9030745 8851159
Kuala Lumpur Malaysia 8980578 8832827
Wuhan China 8951200 8834698
Suzhou China 8588504 8351414
Hangzhou China 8625267 8418349
Surat India 8592860 8338575
Dar es Salaam Tanzania 8529744 8188494
Baghdad Iraq 8154140 7911328
Shenyang China 7944044 7831541
Riyadh Saudi Arabia 7964688 7848751
New York City United States 7966324 8100605
Foshan China 7833467 7694081
Dongguan China 7803482 7693276
Hong Kong Hong Kong 7791531 7716372
Pune India 7498726 7369021
Haerbin China 7082652 6925340
Santiago Chile 6973392 6953542
Madrid Spain 6826620 6800842
Khartoum Sudan 6778168 6526345
Toronto Canada 6513813 6450438
Johannesburg South Africa 6436807 6339743
Belo Horizonte Brazil 6360069 6281752
Dalian China 6360035 6239756
Qingdao China 6242353 6098734
Singapore Singapore 6167759 6115882
Zhengzhou China 6165031 5992273
Ji nan Shandong China 6062368 5922823
Abidjan Ivory Coast 6054358 5859424
Addis Ababa Ethiopia 5961711 5681609
Yangon Myanmar 5829964 5706310
Alexandria Egypt 5801580 5707049
Nairobi Kenya 5772121 5560131
Barcelona Spain 5751075 5730564
Chittagong Bangladesh 5635870 5533483
Hanoi Vietnam 5593577 5421696
Saint Petersburg Russia 5577807 5589909
Guadalajara Mexico 5577341 5496809
Ankara Turkey 5565915 5476444
Fukuoka Japan 5452552 5499374
Melbourne Australia 5404124 5305432
Monterrey Mexico 5288432 5215757
Sydney Australia 5258950 5188593
Urumqi China 5116895 5001934
Changsha China 5112784 5047816
Cape Town South Africa 5064396 4958870
Jiddah Saudi Arabia 5015645 4923708
Brasilia Brazil 4998660 4916085
Kunming China 4937047 4865282
Changchun China 4900896 4796560
Kabul Afghanistan 4862586 4712793
Yaounde Cameroon 4859198 4692347
Hefei China 4822842 4735500
Ningbo China 4780151 4659518
Shantou China 4737079 4651827
Kano Nigeria 4638799 4481282
Tel Aviv Israel 4577871 4500492
New Taipei Taiwan 4570576 4522439
Shijiazhuang China 4527329 4462103
Jaipur India 4406280 4321166
================================================
FILE: tests/geography-a.json
================================================
[
["Tokyo","Japan",37131070,37173748],
["Delhi","India",34598951,33741241],
["Shanghai","China",30365228,29973570],
["Dhaka","Bangladesh",24561693,23958442],
["Cairo","Egypt",23095986,22623336],
["Sao Paulo","Brazil",23045227,22830606],
["Mexico City","Mexico",22831373,22471468],
["Beijing","China",22559204,22155716],
["Mumbai","India",22012722,21690477],
["Osaka","Japan",18990205,19040445],
["Chongqing","China",18100872,17809422],
["Karachi","Pakistan",18059805,17637893],
["Kinshasa","DR Congo",17815364,17025505],
["Lagos","Nigeria",17124998,16554804],
["Istanbul","Turkey",16211581,16054896],
["Kolkata","India",15904385,15535352],
["Buenos Aires","Argentina",15714124,15634092],
["Manila","Philippines",15211511,14917612],
["Guangzhou","China",14870254,14634025],
["Lahore","Pakistan",14835678,14359466],
["Tianjin","China",14729021,14417999],
["Bangalore","India",14387359,14059333],
["Rio de Janeiro","Brazil",13880630,13774171],
["Shenzhen","China",13501772,13330796],
["Moscow","Russia",12768223,12699759],
["Chennai","India",12353002,12009954],
["Bogota","Colombia",11779275,11660428],
["Jakarta","Indonesia",11628728,11478423],
["Lima","Peru",11529982,11388304],
["Bangkok","Thailand",11415533,11195528],
["Paris","France",11352823,11304387],
["Hyderabad","India",11307135,11046182],
["Nanjing","China",10208049,9935614],
["Luanda","Angola",10049628,9648709],
["Seoul","South Korea",10059272,9991484],
["Chengdu","China",10001659,9809118],
["London","United Kingdom",9818142,9723207],
["Ho Chi Minh City","Vietnam",9798896,9541155],
["Tehran","Iran",9738111,9606808],
["Nagoya","Japan",9544065,9544935],
["Xi-an","China",9253706,9000317],
["Ahmedabad","India",9030745,8851159],
["Kuala Lumpur","Malaysia",8980578,8832827],
["Wuhan","China",8951200,8834698],
["Suzhou","China",8588504,8351414],
["Hangzhou","China",8625267,8418349],
["Surat","India",8592860,8338575],
["Dar es Salaam","Tanzania",8529744,8188494],
["Baghdad","Iraq",8154140,7911328],
["Shenyang","China",7944044,7831541],
["Riyadh","Saudi Arabia",7964688,7848751],
["New York City","United States",7966324,8100605],
["Foshan","China",7833467,7694081],
["Dongguan","China",7803482,7693276],
["Hong Kong","Hong Kong",7791531,7716372],
["Pune","India",7498726,7369021],
["Haerbin","China",7082652,6925340],
["Santiago","Chile",6973392,6953542],
["Madrid","Spain",6826620,6800842],
["Khartoum","Sudan",6778168,6526345],
["Toronto","Canada",6513813,6450438],
["Johannesburg","South Africa",6436807,6339743],
["Belo Horizonte","Brazil",6360069,6281752],
["Dalian","China",6360035,6239756],
["Qingdao","China",6242353,6098734],
["Singapore","Singapore",6167759,6115882],
["Zhengzhou","China",6165031,5992273],
["Ji nan Shandong","China",6062368,5922823],
["Abidjan","Ivory Coast",6054358,5859424],
["Addis Ababa","Ethiopia",5961711,5681609],
["Yangon","Myanmar",5829964,5706310],
["Alexandria","Egypt",5801580,5707049],
["Nairobi","Kenya",5772121,5560131],
["Barcelona","Spain",5751075,5730564],
["Chittagong","Bangladesh",5635870,5533483],
["Hanoi","Vietnam",5593577,5421696],
["Saint Petersburg","Russia",5577807,5589909],
["Guadalajara","Mexico",5577341,5496809],
["Ankara","Turkey",5565915,5476444],
["Fukuoka","Japan",5452552,5499374],
["Melbourne","Australia",5404124,5305432],
["Monterrey","Mexico",5288432,5215757],
["Sydney","Australia",5258950,5188593],
["Urumqi","China",5116895,5001934],
["Changsha","China",5112784,5047816],
["Cape Town","South Africa",5064396,4958870],
["Jiddah","Saudi Arabia",5015645,4923708],
["Brasilia","Brazil",4998660,4916085],
["Kunming","China",4937047,4865282],
["Changchun","China",4900896,4796560],
["Kabul","Afghanistan",4862586,4712793],
["Yaounde","Cameroon",4859198,4692347],
["Hefei","China",4822842,4735500],
["Ningbo","China",4780151,4659518],
["Shantou","China",4737079,4651827],
["Kano","Nigeria",4638799,4481282],
["Tel Aviv","Israel",4577871,4500492],
["New Taipei","Taiwan",4570576,4522439],
["Shijiazhuang","China",4527329,4462103],
["Jaipur","India",4406280,4321166]
]
================================================
FILE: tests/hline-hash.json
================================================
[
["bb","cc"],
null,
{"aa":122,"bb":654,"cc":"apple"},
{"aa":34,"bb":-78,"cc":"grape"},
{},
{"aa":12,"bb":"+45","cc": "grape"},
{"aa":24,"bb":"+35","cc": "apple"},
{"aa":33,"bb":"+66","cc": "apple"},
{"aa":10,"bb":"+91" ,"cc": "apple"},
[],
[-15,"apple",122],
{"aa":8 ,"bb": 7,"cc": "grape"},
{"aa":1,"bb": 4 ,"cc": "grape"}
]
================================================
FILE: tests/hline-header.json
================================================
[
["aa","bb","cc"],
null,
[122,654,"apple"],
[34,-78,"grape"],
[],
[12,"+45", "grape"],
[24,"+35", "apple"],
[33,"+66", "apple"],
[10,"+91" , "apple"],
[],
[122,-15, "apple"],
[8 , 7, "grape"],
[1, 4 , "grape"]
]
================================================
FILE: tests/hline.csv
================================================
aa;bb;cc
122;654;apple
34;-78;grape
12;+45; "grape"
24;+35; "apple"
33;+66; "apple"
10;+91 ; "apple"
122,-15, "apple"
8 , 7, "grape"
1, 4 , "grape"
================================================
FILE: tests/unfoldtest.org
================================================
# -*- coding:utf-8; -*-
#+TITLE: Tests for unfolding Orgtbl Aggregate "BEGIN: aggregate" line
Copyright (C) 2013-2026 Thierry Banel
org-aggregate is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
org-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
* How to run?
Running all tests should not change anything to this page.
Run this script to complete all the unit tests in a disposable
buffer. When done, the buffer and the original, untouched
~unfoldtest.org~, are compared, stopping at the first difference.
#+begin_src elisp :results none
;; define a utility to run a unit test
(defun orgtbl-aggregate-unfold-test (&rest args)
(execute-kbd-macro
(kbd
(mapconcat #'identity (cons "C-n" args) "\n"))))
;; display this buffer in a window occupying half the frame
(delete-other-windows)
(goto-char (point-min))
(org-cycle '(64))
(split-window-right)
;; Make a new buffer and fill it with the content of unittests.org
(let ((f (buffer-file-name)))
(switch-to-buffer "disposable-unfoldtests.org")
(erase-buffer)
(insert-file f))
(org-mode)
(org-cycle '(64))
;; Clean results from prior tests:
;; replace a pair of a simple and a complex aggregate blocks
;; with 2 copies of the simple one
(rx-define aggregateblock
(group
"#+begin: " (* not-newline) "\n"
(*? (* any) "\n")
"#+end:" (* any) "\n"))
(save-excursion
(goto-char (point-min))
(replace-regexp
(rx bol aggregateblock aggregateblock)
"\\2\\2"))
;; call all wizard invocations
(goto-char (point-min))
(org-next-visible-heading 2)
;;(while (search-forward "(execute-kbd-macro" nil t)
;; (beginning-of-line)
;; (forward-sexp)
;; (forward-line)
;; (beginning-of-line)
;; (eval-last-sexp nil))
(while (re-search-forward
(rx "orgtbl-aggregate-unfold-test"
(*? (* any) "\n")
"#+end_src")
nil t)
(beginning-of-line)
(org-ctrl-c-ctrl-c))
;; Compare the disposable buffer with the reference
(goto-char (point-min))
(compare-windows nil)
#+end_src
* unua testo
#+begin_src elisp :results none
(orgtbl-aggregate-unfold-test
"C-a "
" ( csv SPC header )"
" [ : ]"
" a a * ; ' b b '"
" C-e v s u m ( b b ) SPC"
" "
" ( l a m b d a SPC ( x ) SPC x )"
" "
"C-c C-c")
#+end_src
#+BEGIN: aggregate :table "hline.csv:(csv header)[0:12]" :precompute "aa*10;'bb'" :cols "vsum(aa) vsum(bb) count()" :hline "1" :post "(lambda (x) x)"
| vsum(aa) | vsum(bb) | count() |
|----------+----------+---------|
| 156 | 576 | 2 |
|----------+----------+---------|
| 79 | 237 | 4 |
|----------+----------+---------|
| 131 | -4 | 3 |
#+END:
#+BEGIN: aggregate :table "hline.csv:" :cols "vsum(aa) count()" :hline "0"
| vsum(aa) | count() |
|----------+---------|
| 366 | 9 |
#+END:
* quote quotes
#+begin_src elisp :results none
(orgtbl-aggregate-unfold-test
"C-a "
"C-s : p o s ( lambda SPC ( table ) SPC ( append SPC table SPC ' ( hline SPC ( \"total\" SPC 0 ) ) ) ) "
"C-r b e g i "
"C-c C-c")
#+end_src
#+BEGIN: aggregate :table "distant-tests.org:distanttable" :cols "tag vsum(val)" :post "(lambda (table) (append table '(hline (\"total\" 0))))"
| tag | vsum(val) |
|-------+-----------|
| A | 405.43 |
| BB | 453.63 |
| CCC | 215.71 |
|-------+-----------|
| total | 0 |
#+END:
#+BEGIN: aggregate :table "distant-tests.org:distanttable" :cols "tag vsum(val)"
| tag | vsum(val) |
|-----+-----------|
| A | 405.43 |
| BB | 453.63 |
| CCC | 215.71 |
#+END:
================================================
FILE: tests/unittests.org
================================================
# -*- coding:utf-8; -*-
#+TITLE: Unit Tests & Examples for Orgtbl Aggregate
Copyright (C) 2013-2026 Thierry Banel
org-aggregate is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
org-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
* How to run?
Running all tests should not change anything to this page.
Run this script to complete all the unit tests in a disposable
buffer. When done, the buffer and the original, untouched
~unittests.org~, are compared, stopping at the first difference.
#+begin_src elisp :results none
(delete-other-windows)
(goto-char (point-min))
(org-cycle '(64))
(split-window-right)
;; Make a new buffer and fill it with the content of unittests.org
(let ((f (buffer-file-name)))
(switch-to-buffer "disposable-unittest.org")
(erase-buffer)
(insert-file f))
(org-mode)
(org-cycle '(64))
;; Clean results from prior tests
(save-excursion
(goto-char (point-min))
(replace-regexp
(rx (group
bol "#+BEGIN" (* not-newline) "\n"
(* (* (any " \t")) "#+" (* not-newline) "\n"))
(* "|" (* not-newline) "\n"))
"\\1\n"))
;; Compute all pull-mode tests
(let ((org-calc-default-modes
(cons 'calc-float-format (cons '(float 12) org-calc-default-modes))))
(org-update-all-dblocks))
;; Compute all push-mode tests
(let ((org-calc-default-modes
(cons 'calc-float-format (cons '(float 12) org-calc-default-modes))))
(org-table-map-tables
(lambda ()
(when (save-excursion
(forward-line -1)
(looking-at-p (rx (or "#+begin" "#+orgtbl"))))
(orgtbl-send-table 'maybe)))))
;; Compare the disposable buffer with the reference unittests.org
(goto-char (point-min))
(compare-windows nil)
#+end_src
* Test PUSH
Push = source table drives computation of aggregated tables.
Run by typing C-c C-c on the first pipe of the source table.
** Source table
No need to name it.
#+ORGTBL: SEND aggtable1 orgtbl-to-aggregated-table :cols "sum($3) $2 sum($4) mean($5) $3*$3 min($5) max($5)"
#+ORGTBL: SEND aggtable2 orgtbl-to-aggregated-table :cols "sum(x) q sum(y) mean(z) x*x min(z) max(z)"
#+ORGTBL: SEND aggtable3 orgtbl-to-aggregated-table :cols "p count() sum($4) mean(z) sum(z*z) (x) min(y) max(y)"
#+ORGTBL: SEND aggtable4 orgtbl-to-aggregated-table :cols "count() mean(x) mean(y) mean(z) meane(z) median(z)" :cond (not (equal f ""))
#+ORGTBL: SEND aggtable5 orgtbl-to-aggregated-table :cols "count() mean(x) mean(y) mean(z) meane(z) median(z) hline"
#+ORGTBL: SEND aggtable6 orgtbl-to-aggregated-table :cols "q prod(z) sdev(z) pvar(z) psdev(z)"
#+ORGTBL: SEND aggtable7 orgtbl-to-aggregated-table :cols "q prod(z) cov(x,y) corr(z,z)"
#+ORGTBL: SEND aggtable8 orgtbl-to-aggregated-table :cols "hline min(d) max(d) mean(d)"
#+ORGTBL: SEND aggtable9 orgtbl-to-aggregated-table :cols "sum(x) q sum(y) mean(z) x*x min(z) max(z)" :cond (equal hline 2)
#+ORGTBL: SEND aggtablea orgtbl-to-aggregated-table :cols "sum(x) q sum(y) mean(z) x*x min(z) max(z)" :cond (equal q "b")
| p | q | x | y | z | f | d |
|---+---+-----+------+---+---+------------------------|
| 1 | b | 12 | 8 | 9 | 0 | [2013-12-22 sun 09:01] |
| 3 | b | 12 | 8 | 9 | 0 | [2013-11-23 sat 13:04] |
| 1 | a | 3 | 2 | 4 | 0 | [2011-09-24 sat 13:54] |
| 2 | a | 3 | 2 | 2 | | [2013-09-25 wed 03:54] |
| 3 | a | 3 | 2 | 1 | | [2014-02-26 wed 16:11] |
| 3 | a | 5 | 4 | 2 | | [2014-01-18 sat 03:51] |
| 1 | a | 5.1 | 2 | 8 | 1 | [2013-12-25 wed 00:00] |
|---+---+-----+------+---+---+------------------------|
| 2 | b | 9 | 8 | 5 | | [2012-12-25 tue 00:00] |
| 3 | b | 9 | 8 | 1 | | [2014-01-18 sat 23:22] |
| 4 | a | a | a | 8 | | [2014-08-02 sat 23:22] |
| 5 | a | a | 10*a | 4 | | [2015-09-14 mon 13:07] |
|---+---+-----+------+---+---+------------------------|
| 4 | b | b | b | 6 | 1 | [2015-10-02 fri 17:42] |
| 5 | b | b+3 | b*b | 8 | 1 | [2016-01-28 thu 15:06] |
** Resulting tables
#+BEGIN RECEIVE ORGTBL aggtable1
| sum($3) | $2 | sum($4) | mean($5) | $3*$3 | min($5) | max($5) |
|------------+----+--------------+---------------+-------------------+---------+---------|
| 2 b + 45 | b | b + b^2 + 32 | 6.33333333333 | 2 b^2 + 6 b + 459 | 1 | 9 |
| 2 a + 19.1 | a | 11 a + 12 | 4.14285714286 | 2 a^2 + 78.01 | 1 | 8 |
#+END RECEIVE ORGTBL aggtable1
#+BEGIN RECEIVE ORGTBL aggtable2
| sum(x) | q | sum(y) | mean(z) | x*x | min(z) | max(z) |
|------------+---+--------------+---------------+-------------------+--------+--------|
| 2 b + 45 | b | b + b^2 + 32 | 6.33333333333 | 2 b^2 + 6 b + 459 | 1 | 9 |
| 2 a + 19.1 | a | 11 a + 12 | 4.14285714286 | 2 a^2 + 78.01 | 1 | 8 |
#+END RECEIVE ORGTBL aggtable2
#+BEGIN RECEIVE ORGTBL aggtable3
| p | count() | sum($4) | mean(z) | sum(z*z) | (x) | min(y) | max(y) |
|---+---------+------------+---------+----------+---------------+----------------+----------------|
| 1 | 3 | 12 | 7 | 161 | [12, 3, 5.1] | 2 | 8 |
| 3 | 4 | 22 | 3.25 | 87 | [12, 3, 5, 9] | 2 | 8 |
| 2 | 2 | 10 | 3.5 | 29 | [3, 9] | 2 | 8 |
| 4 | 2 | a + b | 7 | 100 | [a, b] | min(a, b) | max(a, b) |
| 5 | 2 | 10 a + b^2 | 6 | 80 | [a, b + 3] | min(10 a, b^2) | max(10 a, b^2) |
#+END RECEIVE ORGTBL aggtable3
#+BEGIN RECEIVE ORGTBL aggtable4
| count() | mean(x) | mean(y) | mean(z) | meane(z) | median(z) |
|---------+-------------------------+---------------------------------+---------------+----------------------------------+-----------|
| 6 | 0.333333333333 b + 5.85 | b / 6 + b^2 / 6 + 3.33333333333 | 7.33333333333 | 7.33333333333 +/- 0.802772971919 | 8 |
#+END RECEIVE ORGTBL aggtable4
#+BEGIN RECEIVE ORGTBL aggtable5
| count() | mean(x) | mean(y) | mean(z) | meane(z) | median(z) | hline |
|---------+---------------+-----------------+---------+-----------------------+-----------+-------|
| 7 | 6.15714285714 | 4 | 5 | 5 +/- 1.34518541827 | 4 | 0 |
| 4 | 0.5 a + 4.5 | 2.75 a + 4 | 4.5 | 4.5 +/- 1.44337567297 | 4.5 | 1 |
| 2 | b + 1.5 | b / 2 + b^2 / 2 | 7 | 7 +/- 1 | 7 | 2 |
#+END RECEIVE ORGTBL aggtable5
#+BEGIN RECEIVE ORGTBL aggtable6
| q | prod(z) | sdev(z) | pvar(z) | psdev(z) |
|---+---------+---------------+---------------+---------------|
| b | 19440 | 3.07679486912 | 7.88888888889 | 2.80871659106 |
| a | 4096 | 2.85356919364 | 6.97959183673 | 2.64189171556 |
#+END RECEIVE ORGTBL aggtable6
#+BEGIN RECEIVE ORGTBL aggtable7
| q | prod(z) | cov(x,y) | corr(z,z) |
|---+---------+------------------------------------------------------------------+-----------|
| b | 19440 | 0.133333333333 b^3 - 3.63333333333 b - 0.766666666667 b^2 + 19.2 | 1. |
| a | 4096 | 1.30952380953 a^2 - 5.57380952381 a + 2.5761904762 | 1. |
#+END RECEIVE ORGTBL aggtable7
#+BEGIN RECEIVE ORGTBL aggtable8
| hline | min(d) | max(d) | mean(d) |
|-------+------------------------+------------------------+-----------------------------|
| 0 | <2011-09-24 Sat 13:54> | <2014-02-26 Wed 16:11> | <14089-07-11 Mon 11:55> / 7 |
| 1 | <2012-12-25 Tue 00:00> | <2015-09-14 Mon 13:07> | 735354.373438 |
| 2 | <2015-10-02 Fri 17:42> | <2016-01-28 Thu 15:06> | 735932.683334 |
#+END RECEIVE ORGTBL aggtable8
#+BEGIN RECEIVE ORGTBL aggtable9
| sum(x) | q | sum(y) | mean(z) | x*x | min(z) | max(z) |
|---------+---+---------+---------+-----------------+--------+--------|
| 2 b + 3 | b | b + b^2 | 7 | 2 b^2 + 6 b + 9 | 6 | 8 |
#+END RECEIVE ORGTBL aggtable9
#+BEGIN RECEIVE ORGTBL aggtablea
| sum(x) | q | sum(y) | mean(z) | x*x | min(z) | max(z) |
|----------+---+--------------+---------------+-------------------+--------+--------|
| 2 b + 45 | b | b + b^2 + 32 | 6.33333333333 | 2 b^2 + 6 b + 459 | 1 | 9 |
#+END RECEIVE ORGTBL aggtablea
* Test PULL
Pull = aggregated table knows how to compute itself,
source table is unaware of the aggregation.
** Source table
Not changed in any way by the aggregate process.
(Note: non-ascii characters are used as column names)
#+TBLNAME: pulledtable
| pé | qû | xà | yÿ | zö | déf |
|----+----+-----+------+----+-----|
| 1 | b | 12 | 8 | 9 | |
| 3 | b | 12 | 8 | 9 | |
| 1 | a | 3 | 2 | 4 | 1 |
| 2 | a | 3 | 2 | 2 | |
| 3 | a | 3 | 2 | 1 | 1 |
| 3 | a | 5 | 4 | 2 | 1 |
| 1 | a | 5.1 | 2 | 8 | 1 |
| 2 | b | 9 | 8 | 5 | |
| 3 | b | 9 | 8 | 1 | |
| 4 | a | a | a | 8 | |
| 5 | a | a | 10*a | 4 | 1 |
| 4 | b | b | b | 6 | 1 |
| 5 | b | b+3 | b*b | 8 | |
** Resulting tables
Type C-c C-c within each to refresh
Note the =:formula= parameter to add a new column after the aggregation has been computed.
#+BEGIN: aggregate :table pulledtable :cols ("qû" "mean(zö)") :formula "$3=$2*100"
| qû | mean(zö) | |
|----+---------------+-----------|
| b | 6.33333333333 | 633.33333 |
| a | 4.14285714286 | 414.28571 |
#+TBLFM: $3=$2*100
#+END
Note the additional =$8= column automatically computed after the aggregation
#+BEGIN: aggregate :table pulledtable :cols "sum(xà) qû sum(yÿ) mean(zö) xà*xà min(zö) max(zö)"
| sum(xà) | qû | sum(yÿ) | mean(zö) | xà*xà | min(zö) | max(zö) | |
|------------+----+--------------+---------------+-------------------+---------+---------+-----|
| 2 b + 45 | b | b + b^2 + 32 | 6.33333333333 | 2 b^2 + 6 b + 459 | 1 | 9 | 5 |
| 2 a + 19.1 | a | 11 a + 12 | 4.14285714286 | 2 a^2 + 78.01 | 1 | 8 | 4.5 |
#+TBLFM: $8=($6+$7)/2
#+END
#+BEGIN: aggregate :table pulledtable :cols "pé count() sum($4) mean(zö) sum(zö*zö) (xà) min(yÿ) max(yÿ)"
#+caption: named_table
#+attr_latex: :environment longtable :width \linewidth
| pé | count() | sum($4) | mean(zö) | sum(zö*zö) | (xà) | min(yÿ) | max(yÿ) |
|----+---------+------------+----------+------------+---------------+----------------+----------------|
| 1 | 3 | 12 | 7 | 161 | [12, 3, 5.1] | 2 | 8 |
| 3 | 4 | 22 | 3.25 | 87 | [12, 3, 5, 9] | 2 | 8 |
| 2 | 2 | 10 | 3.5 | 29 | [3, 9] | 2 | 8 |
| 4 | 2 | a + b | 7 | 100 | [a, b] | min(a, b) | max(a, b) |
| 5 | 2 | 10 a + b^2 | 6 | 80 | [a, b + 3] | min(10 a, b^2) | max(10 a, b^2) |
#+END
#+BEGIN: aggregate :table pulledtable :cols "count() mean(xà) mean(yÿ) mean(zö)"
| count() | mean(xà) | mean(yÿ) | mean(zö) |
|---------+-----------------------------------------------------+------------------------------------------------------+---------------|
| 13 | 0.153846153846 a + 0.153846153846 b + 4.93076923077 | 0.846153846154 a + b / 13 + b^2 / 13 + 3.38461538462 | 5.15384615385 |
#+END
#+BEGIN: aggregate :table pulledtable :cols "pé count() mean(zö) meane(zö) gmean(zö) hmean(zö) median(zö)"
| pé | count() | mean(zö) | meane(zö) | gmean(zö) | hmean(zö) | median(zö) |
|----+---------+----------+------------------------+---------------+---------------+------------|
| 1 | 3 | 7 | 7 +/- 1.52752523165 | 6.60385449779 | 6.17142857143 | 8 |
| 3 | 4 | 3.25 | 3.25 +/- 1.93110503771 | 2.05976714391 | 1.53191489362 | 1.5 |
| 2 | 2 | 3.5 | 3.5 +/- 1.5 | 3.16227766017 | 2.85714285714 | 3.5 |
| 4 | 2 | 7 | 7 +/- 1 | 6.92820323028 | 6.85714285714 | 7 |
| 5 | 2 | 6 | 6 +/- 2 | 5.65685424949 | 5.33333333333 | 6 |
#+END
#+BEGIN: aggregate :table pulledtable :cols "qû count() prod(zö) sdev(zö) pvar(zö) psdev(zö)"
| qû | count() | prod(zö) | sdev(zö) | pvar(zö) | psdev(zö) |
|----+---------+----------+---------------+---------------+---------------|
| b | 6 | 19440 | 3.07679486912 | 7.88888888889 | 2.80871659106 |
| a | 7 | 4096 | 2.85356919364 | 6.97959183673 | 2.64189171556 |
#+END
#+BEGIN: aggregate :table pulledtable :cols "qû count() cov(zö,xà) pcov(zö,zö) corr(zö,zö)"
| qû | count() | cov(zö,xà) | pcov(zö,zö) | corr(zö,zö) |
|----+---------+----------------------------------+---------------+-------------|
| b | 6 | 0.266666666666 b + 1.8 | 7.88888888889 | 1. |
| a | 7 | 0.619047619047 a - 1.22142857142 | 6.97959183673 | 1. |
#+END
* Test :cond PUSH
** Source table
Only the second group (5 rows) is considered with the test =hline=1=.
#+ORGTBL: SEND aggtable15 orgtbl-to-aggregated-table :cond (equal hline 1) :cols "count() q mean(x) mean(y) mean(z) hline"
| p | q | x | y | z |
|---+---+-----+------+---|
| 1 | b | 12 | 8 | 9 |
| 3 | b | 12 | 8 | 9 |
| 1 | a | 3 | 2 | 4 |
| 2 | a | 3 | 2 | 2 |
| 3 | a | 3 | 2 | 1 |
| 3 | a | 5 | 4 | 2 |
| 1 | a | 5.1 | 2 | 8 |
|---+---+-----+------+---|
| 2 | b | 9 | 8 | 5 |
| 3 | b | 9 | 8 | 1 |
| 4 | a | a | a | 8 |
| 5 | a | a | 10*a | 4 |
| 4 | b | b | b | 6 |
|---+---+-----+------+---|
| 5 | b | b+3 | b*b | 8 |
** Aggregated table
#+BEGIN RECEIVE ORGTBL aggtable15
| count() | q | mean(x) | mean(y) | mean(z) | hline |
|---------+---+-----------+-----------------------+---------+-------|
| 3 | b | b / 3 + 6 | b / 3 + 5.33333333333 | 4 | 1 |
| 2 | a | a | 5.5 a | 6 | 1 |
#+END RECEIVE ORGTBL aggtable15
* Test :cond PULL
The =:cond= parameter takes a lisp expression
to filter-out resulting rows.
** Resulting tables
Only consider rows for which column q have the value "b"
#+BEGIN: aggregate :table pulledtable :cols "qû count() mean(zö)" :cond (equal qû "b")
| qû | count() | mean(zö) |
|----+---------+---------------|
| b | 6 | 6.33333333333 |
#+END
Only consider rows for which column =p= is greater than =3=.
Note the =string-to-number= call, because cells always contain strings.
#+BEGIN: aggregate :table pulledtable :cols "qû count() mean(zö)" :cond (>= (string-to-number pé) 3)
| qû | count() | mean(zö) |
|----+---------+----------|
| b | 4 | 6 |
| a | 4 | 3.75 |
#+END
Only consider rows for which the =def= column is not blank.
#+BEGIN: aggregate :table pulledtable :cols "qû count() mean(zö) déf" :cond (not (equal déf ""))
| qû | count() | mean(zö) | déf |
|----+---------+----------+-----|
| a | 5 | 3.8 | 1 |
| b | 1 | 6 | 1 |
#+END
* Test correlation
Are two columns correlated ?
** Source table
Contains columns correlated with some noise.
: y = 10* + noise (x y are highly correlated)
: z = pure noise (x z are not correlated)
: t = pure noise (z t are not correlated)
: m = 10*x in reverse order (x m are negative correlated)
#+TBLNAME: correlated
| tag | x | y | z | t | m |
|-------+----+---------+-------+-------+-----|
| small | 1 | 10.414 | 78.30 | 1.70 | 120 |
| small | 2 | 20.616 | 48.20 | 80.40 | 110 |
| small | 3 | 30.210 | 93.50 | 25.10 | 100 |
| small | 4 | 41.692 | 85.90 | 16.30 | 90 |
| small | 5 | 50.576 | 11.70 | 37.00 | 80 |
| large | 6 | 60.026 | 46.60 | 6.00 | 70 |
| large | 7 | 71.236 | 3.30 | 35.70 | 60 |
| large | 8 | 81.204 | 78.80 | 46.30 | 50 |
| large | 9 | 90.862 | 89.60 | 98.40 | 40 |
| large | 10 | 101.240 | 0.60 | 8.80 | 30 |
| large | 11 | 111.924 | 32.40 | 63.70 | 20 |
| large | 12 | 120.490 | 35.50 | 98.20 | 10 |
The following line was appended to the table to generate the random noise.
It is thrown away to avoid recomputing new noise, and thus invalidating the test.
: #+TBLFM: $3=$2*10+random(1000)/500;%.3f::$4=random(1000)/10;%.2f::$5=random(1000)/10;%.2f
** Resulting table
Type C-c C-c within resulting table to refresh.
#+BEGIN: aggregate :table correlated :cols "tag corr(x,y) corr(x,z) corr(x,m) corr(z,t)"
| tag | corr(x,y) | corr(x,z) | corr(x,m) | corr(z,t) |
|-------+----------------+-----------------+-----------+----------------|
| small | 0.999449791325 | -0.448296141593 | -1 | -0.49786310458 |
| large | 0.999657841285 | -0.120566390616 | -1 | 0.486014333463 |
#+END
* Test without headers
What if the source table does not have headers?
Then columns should be named =$1=, =$2=, =$3= and so on.
** Source table
#+TBLNAME: noheader
| 0 | z | t | x | y |
| 1 | b | 12 | 8 | 9 |
| 3 | b | 12 | 8 | 9 |
| 1 | a | 3 | 2 | 4 |
| 2 | a | 3 | 2 | 2 |
| 3 | a | 3 | 2 | 1 |
| 3 | a | 5 | 4 | 2 |
| 1 | a | 5.1 | 2 | 8 |
| 2 | b | 9 | 8 | 5 |
| 3 | b | 9 | 8 | 1 |
| 4 | a | a | a | 8 |
| 5 | a | a | 10*a | 4 |
| 4 | b | b | b | 6 |
| 5 | b | b+3 | b*b | 8 |
** Aggregated table
#+BEGIN: aggregate :table noheader :cols "hline $1 mean($3) sum($4)"
| hline | $1 | mean($3) | sum($4) |
|-------+----+---------------------+------------|
| 0 | 0 | t | x |
| 0 | 1 | 6.7 | 12 |
| 0 | 3 | 7.25 | 22 |
| 0 | 2 | 6 | 10 |
| 0 | 4 | a / 2 + b / 2 | a + b |
| 0 | 5 | a / 2 + b / 2 + 1.5 | 10 a + b^2 |
#+END
* Test hline grouping
Horizontal lines naturally create groups withing the source table.
Those groups can be accessed through the =hline= virtual column.
** Source table
It contains four groups separated by horizontal lines.
#+TBLNAME: hlinetable
| p | q | x | y | z | f |
|---+---+-----+------+---+---|
| 1 | b | 12 | 8 | 9 | 0 |
| 3 | b | 12 | 8 | 9 | 0 |
| 1 | a | 3 | 2 | 4 | 0 |
| 2 | a | 3 | 2 | 2 | 0 |
| 3 | a | 3 | 2 | 1 | 0 |
|---+---+-----+------+---+---|
| 3 | a | 5 | 4 | 2 | 1 |
| 1 | a | 5.1 | 2 | 8 | 1 |
|---+---+-----+------+---+---|
| 2 | b | 9 | 8 | 5 | 1 |
| 3 | b | 9 | 8 | 1 | 1 |
| 4 | a | a | a | 8 | 1 |
|---+---+-----+------+---+---|
| 5 | a | a | 10*a | 4 | 1 |
| 4 | b | b | b | 6 | 1 |
| 5 | b | b+3 | b*b | 8 | 1 |
** Aggregated table
The =hline= column groups data
#+BEGIN: aggregate :table hlinetable :cols "q hline vcount()" :cond (equal f "1")
| q | hline | vcount() |
|---+-------+----------|
| a | 1 | 2 |
| b | 2 | 2 |
| a | 2 | 1 |
| a | 3 | 1 |
| b | 3 | 2 |
#+END
* Test @# row numbering
are a's & b's near the beginning or the end of the input table?
#+BEGIN: aggregate :table "hlinetable" :cols "q vmean(@#)"
| q | vmean(@#) |
|---+---------------|
| b | 9.16666666667 |
| a | 7.57142857143 |
#+END:
p & q columns in reverse order
#+BEGIN: aggregate :table "hlinetable" :cols "p q hline @#;^N2;<>"
| p | q | hline |
|---+---+-------|
| 5 | b | 3 |
| 4 | b | 3 |
| 5 | a | 3 |
| 4 | a | 2 |
| 3 | b | 2 |
| 2 | b | 2 |
| 1 | a | 1 |
| 3 | a | 1 |
| 3 | a | 0 |
| 2 | a | 0 |
| 1 | a | 0 |
| 3 | b | 0 |
| 1 | b | 0 |
#+END:
#+BEGIN: aggregate :table "hlinetable" :cols "p q hline @#;^N2"
| p | q | hline | @# |
|---+---+-------+----|
| 5 | b | 3 | 16 |
| 4 | b | 3 | 15 |
| 5 | a | 3 | 14 |
| 4 | a | 2 | 12 |
| 3 | b | 2 | 11 |
| 2 | b | 2 | 10 |
| 1 | a | 1 | 8 |
| 3 | a | 1 | 7 |
| 3 | a | 0 | 5 |
| 2 | a | 0 | 4 |
| 1 | a | 0 | 3 |
| 3 | b | 0 | 2 |
| 1 | b | 0 | 1 |
#+END:
#+BEGIN: transpose :table "hlinetable" :cols "p q @# hline z f x y"
| p | | 1 | 3 | 1 | 2 | 3 | | 3 | 1 | | 2 | 3 | 4 | | 5 | 4 | 5 |
| q | | b | b | a | a | a | | a | a | | b | b | a | | a | b | b |
| 0 | | 2 | 3 | 4 | 5 | 6 | | 8 | 9 | | 11 | 12 | 13 | | 15 | 16 | 17 |
| 0 | | 1 | 1 | 1 | 1 | 1 | | 2 | 2 | | 3 | 3 | 3 | | 4 | 4 | 4 |
| z | | 9 | 9 | 4 | 2 | 1 | | 2 | 8 | | 5 | 1 | 8 | | 4 | 6 | 8 |
| f | | 0 | 0 | 0 | 0 | 0 | | 1 | 1 | | 1 | 1 | 1 | | 1 | 1 | 1 |
| x | | 12 | 12 | 3 | 3 | 3 | | 5 | 5.1 | | 9 | 9 | a | | a | b | b+3 |
| y | | 8 | 8 | 2 | 2 | 2 | | 4 | 2 | | 8 | 8 | a | | 10*a | b | b*b |
#+END:
#+BEGIN: transpose :table "hlinetable" :cols "p q @# hline z f x y"
#+name: thetransposed
| p | | 1 | 3 | 1 | 2 | 3 | | 3 | 1 | | 2 | 3 | 4 | | 5 | 4 | 5 | 10 |
| q | | b | b | a | a | a | | a | a | | b | b | a | | a | b | b | 10 b |
| 0 | | 2 | 3 | 4 | 5 | 6 | | 8 | 9 | | 11 | 12 | 13 | | 15 | 16 | 17 | 20 |
| 0 | | 1 | 1 | 1 | 1 | 1 | | 2 | 2 | | 3 | 3 | 3 | | 4 | 4 | 4 | 10 |
| z | | 9 | 9 | 4 | 2 | 1 | | 2 | 8 | | 5 | 1 | 8 | | 4 | 6 | 8 | 90 |
| f | | 0 | 0 | 0 | 0 | 0 | | 1 | 1 | | 1 | 1 | 1 | | 1 | 1 | 1 | 0 |
| x | | 12 | 12 | 3 | 3 | 3 | | 5 | 5.1 | | 9 | 9 | a | | a | b | b+3 | 120 |
| y | | 8 | 8 | 2 | 2 | 2 | | 4 | 2 | | 8 | 8 | a | | 10*a | b | b*b | 80 |
#+TBLFM: $19=$3*10
#+END:
* Test dates [YYYY-MM-DD day. HH:MM] style
Some (limited) handling of dates is available.
** Source table
#+tblname: datetable
| n | d |
|---+-------------------------|
| 1 | [2013-12-22 dim. 09:01] |
| 2 | [2013-11-23 sam. 13:04] |
| 3 | [2011-09-24 sam. 13:54] |
| 4 | [2013-09-25 mer. 03:54] |
| 5 | [2014-02-26 mer. 16:11] |
| 6 | [2014-01-18 sam. 03:51] |
| 7 | [2013-12-25 mer. 00:00] |
| 8 | [2012-12-25 mar. 00:00] |
** Aggregated table
#+BEGIN: aggregate :table datetable :cols "min(d) max(d) min(n) max(n) mean(d)"
| min(d) | max(d) | min(n) | max(n) | mean(d) |
|------------------------+------------------------+--------+--------+---------------|
| <2011-09-24 Sat 13:54> | <2014-02-26 Wed 16:11> | 1 | 8 | 735073.937066 |
#+END
* Test durations HH:MM:SS style
** Source table
#+name: some_durations
| dur |
|----------|
| 07:45:30 |
| 13:55 |
| 17:12 |
#+name: some_durations_in_different_formats
| dur |
|----------|
| 01:30:01 |
| 1:30:02 |
| 01:30 |
| 1:30 |
| 100:30 |
** Aggregated table
Test T, U, t formatters
#+BEGIN: aggregate :table "some_durations" :cols "vmean(dur) vmean(dur);T vmean(dur);t vmean(dur);U"
| vmean(dur) | vmean(dur) | vmean(dur) | vmean(dur) |
|------------+------------+------------+------------|
| 46650 | 12:57:30 | 12.96 | 12:57 |
#+END:
#+BEGIN: aggregate :table "some_durations_in_different_formats" :cols "vsum(dur);T"
| vsum(dur) |
|-----------|
| 106:30:03 |
#+END
* Test durations HH@ MM' SS" style
#+name: calc_durations
| dur |
|------------|
| 07@ 45' 30 |
| 13@ 55' |
| 17@ 12' |
#+BEGIN: aggregate :table "calc_durations" :cols "vmean(dur)"
| vmean(dur) |
|--------------|
| 12@ 57' 30." |
#+END:
* Test symbolic
The Emacs Calc symbolic calculator is used by the aggregate package.
Therefore, symbolic calculations are available.
** Source table
Contains the variables =x= and =a=, which are not numeric.
#+TBLNAME: symtable
| Day | Color | Level | Quantity |
|-----------+-------+--------+----------|
| Monday | Red | 30+x | 11+a |
| Monday | Blue | 25+3*x | 3 |
| Thuesday | Red | 51+2*x | 12 |
| Thuesday | Red | 45-x | 15 |
| Thuesday | Blue | 33 | 18 |
| Wednesday | Red | 27 | 23 |
| Wednesday | Blue | 12+x | 16 |
| Wednesday | Blue | 15 | 15-6*a |
| Turdsday | Red | 39 | 24-5*a |
| Turdsday | Red | 41 | 29 |
| Turdsday | Red | 49+x | 30+9*a |
| Friday | Blue | 7 | 5+a |
| Friday | Blue | 6 | 8 |
| Friday | Blue | 11 | 9 |
** Aggregated table
Result is variabilized with =x= and =a=.
#+BEGIN: aggregate :table "symtable" :cols "Day mean(Level) sum(Quantity)"
| Day | mean(Level) | sum(Quantity) |
|-----------+-------------+---------------|
| Monday | 2 x + 27.5 | a + 14 |
| Thuesday | x / 3 + 43 | 45 |
| Wednesday | x / 3 + 18 | 54 - 6 a |
| Turdsday | x / 3 + 43. | 4 a + 83 |
| Friday | 8 | a + 22 |
#+END
* Test zero output
The following test produces sums which happen to be zero, either
because input is empty, or by chance (1-1 = 0).
Zeros are no longer translated to empty cells.
#+TBLNAME: resultzero
| Item | Value |
|------+-------|
| a2 | 1 |
| a2 | 1 |
| a0 | -1 |
| a0 | 1 |
| b2 | 2 |
| b2 | |
| b0 | 0 |
| b0 | |
| c | |
| c | |
#+BEGIN: aggregate :table resultzero :cols "Item vsum(Value) vmean(Value)"
| Item | vsum(Value) | vmean(Value) |
|------+-------------+--------------|
| a2 | 2 | 1 |
| a0 | 0 | 0 |
| b2 | 2 | 2 |
| b0 | 0 | 0 |
| c | 0 | vmean([]) |
#+END
* Test empty inputs
Empty input cells are most often ignored.
- This makes no difference for =sum= and =count=.
- For =prod=, empty input do not result in zero.
- For =mean=, only non-empty cells participate
(if empty cells were zero, they would count in the division).
- For =min= and =max=, a possibly empty list of values is possible,
resulting in =inf= or =-inf=
Some aggregation functions operate on two columns.
In this case, a pair of empty cells is ignored.
But a pair of an empty and a non-empty cell is
added to the aggregation, by replacing the missing
value with zero.
#+tblname: emptyinput
| T | Q | R |
|------------------+----+-----|
| no-blank | 1 | 10 |
| no-blank | 2 | 20 |
| no-blank | 3 | 30 |
| 1-left-blank | 4 | 40 |
| 1-left-blank | | 50 |
| 1-left-blank | 6 | 60 |
| 1-left-blank | 7 | 70 |
| all-blank | | |
| all-blank | | |
| all-blank | | |
| 2-left-blank | 11 | 110 |
| 2-left-blank | 12 | 120 |
| 2-left-blank | 13 | 130 |
| 2-left-blank | 14 | 140 |
| 1-dual-blank | 15 | 150 |
| 1-dual-blank | | |
| 1-dual-blank | 17 | 170 |
| single-non-blank | 18 | 180 |
| single-non-blank | | |
| single-non-blank | | |
#+BEGIN: aggregate :table "emptyinput" :cols "T sum(Q) prod(Q) (Q) min(Q) max(Q)"
| T | sum(Q) | prod(Q) | (Q) | min(Q) | max(Q) |
|------------------+--------+---------+------------------+--------+--------|
| no-blank | 6 | 6 | [1, 2, 3] | 1 | 3 |
| 1-left-blank | 17 | 168 | [4, 6, 7] | 4 | 7 |
| all-blank | 0 | 1 | [] | inf | -inf |
| 2-left-blank | 50 | 24024 | [11, 12, 13, 14] | 11 | 14 |
| 1-dual-blank | 32 | 255 | [15, 17] | 15 | 17 |
| single-non-blank | 18 | 18 | [18] | 18 | 18 |
#+END:
#+BEGIN: aggregate :table "emptyinput" :cols "T mean(Q) meane(Q) gmean(Q) hmean(Q)"
| T | mean(Q) | meane(Q) | gmean(Q) | hmean(Q) |
|------------------+---------------+----------------------------------+---------------+---------------|
| no-blank | 2 | 2 +/- 0.577350269189 | 1.81712059283 | 1.63636363636 |
| 1-left-blank | 5.66666666667 | 5.66666666667 +/- 0.881917103688 | 5.51784835276 | 5.36170212766 |
| all-blank | vmean([]) | vmeane([]) | vgmean([]) | vhmean([]) |
| 2-left-blank | 12.5 | 12.5 +/- 0.645497224368 | 12.4497700445 | 12.399483871 |
| 1-dual-blank | 16 | 16 +/- 1 | 15.9687194227 | 15.9375 |
| single-non-blank | 18 | vmeane([18]) | 18 | 18. |
#+END:
#+BEGIN: aggregate :table "emptyinput" :cols "T min(Q) max(Q)"
| T | min(Q) | max(Q) |
|------------------+--------+--------|
| no-blank | 1 | 3 |
| 1-left-blank | 4 | 7 |
| all-blank | inf | -inf |
| 2-left-blank | 11 | 14 |
| 1-dual-blank | 15 | 17 |
| single-non-blank | 18 | 18 |
#+END:
#+BEGIN: aggregate :table "emptyinput" :cols "T pvar(Q) sdev(Q) psdev(Q)"
| T | pvar(Q) | sdev(Q) | psdev(Q) |
|------------------+----------------+---------------+----------------|
| no-blank | 0.666666666667 | 1 | 0.816496580928 |
| 1-left-blank | 1.55555555556 | 1.52752523165 | 1.24721912893 |
| all-blank | vpvar([]) | vsdev([]) | vpsdev([]) |
| 2-left-blank | 1.25 | 1.29099444874 | 1.11803398875 |
| 1-dual-blank | 1 | 1.41421356237 | 1 |
| single-non-blank | 0 | vsdev([18]) | 0 |
#+END:
#+BEGIN: aggregate :table "emptyinput" :cols "T corr(Q,R);EN cov(Q,R);EN pcov(Q,R);EN"
| T | corr(Q,R) | cov(Q,R) | pcov(Q,R) |
|------------------+-----------------------------+---------------+---------------|
| no-blank | 1 | 10 | 6.66666666667 |
| 1-left-blank | 0.625543242171 | 25. | 18.75 |
| all-blank | vcorr([0, 0, 0], [0, 0, 0]) | 0 | 0 |
| 2-left-blank | 1. | 16.6666666667 | 12.5 |
| 1-dual-blank | 1. | 863.333333333 | 575.555555556 |
| single-non-blank | 1 | 1080 | 720 |
#+END:
#+BEGIN: aggregate :table "emptyinput" :cols "T count() (Q) (R)"
| T | count() | (Q) | (R) |
|------------------+---------+------------------+----------------------|
| no-blank | 3 | [1, 2, 3] | [10, 20, 30] |
| 1-left-blank | 4 | [4, 6, 7] | [40, 50, 60, 70] |
| all-blank | 3 | [] | [] |
| 2-left-blank | 4 | [11, 12, 13, 14] | [110, 120, 130, 140] |
| 1-dual-blank | 3 | [15, 17] | [150, 170] |
| single-non-blank | 3 | [18] | [180] |
#+END:
* Test empty and non-numeric
#+tblname: nonnumeric
| X |
|----|
| 1 |
| 2 |
| aa |
| |
| 4 |
#+BEGIN: aggregate :table "nonnumeric" :cols "(X) (X);E (X);N (X);EN"
| (X) | (X) | (X) | (X) |
|---------------+--------------------+--------------+-----------------|
| [1, 2, aa, 4] | [1, 2, aa, nan, 4] | [1, 2, 0, 4] | [1, 2, 0, 0, 4] |
#+END:
#+BEGIN: aggregate :table "nonnumeric" :cols "mean(X) mean(X);E mean(X);N mean(X);EN"
| mean(X) | mean(X) | mean(X) | mean(X) |
|---------------+---------+---------+---------|
| aa / 4 + 1.75 | nan | 1.75 | 1.4 |
#+END:
Comparison with the spreadsheet:
| 1 | 1 |
| 2 | 2 |
| aa | aa |
| | |
| 4 | 4 |
|--------------------+-------------------|
| [1, 2, aa, 4] | 0.75 + aa / 4 + 1 |
| [1, 2, aa, nan, 4] | nan |
| [1, 2, 0, 4] | 1.75 |
| [1, 2, 0, 0, 4] | 1.4 |
#+TBLFM: @6$1=@1..@5 :: @7$1=@1..@5;E :: @8$1=@1..@5;N :: @9$1=@1..@5;EN :: @6$2=vmean(@1..@5) :: @7$2=vmean(@1..@5);E :: @8$2=vmean(@1..@5);N :: @9$2=vmean(@1..@5);EN
* Test input errors
#+tblname: inputerrors
| A | Q | R | Z | D |
|---+----+-------+-----------+--------------|
| a | 3 | 10 | 2.3025851 | [2014-11-05] |
| a | 4+ | 20 | 2.9957323 | [2014-11-21] |
| b | t | (88*) | #ERROR | [2014-12-07] |
| b | 1 | 41 | 3.7135721 | [2014-12-23] |
| b | 2 | 111 | 4.7095302 | [2015-01-08] |
| c | 8 | z ' | #ERROR | |
| c | 4= | 4 | 1.3862944 | |
#+TBLFM: $4=log($3)
#+BEGIN: aggregate :table "inputerrors" :cols "A sum(Q) sum(R)"
| A | sum(Q) | sum(R) |
|---+------------------------------------+--------------------------------------|
| a | error(2, '"Expected a number") + 3 | 30 |
| b | t + 3 | error(4, '"Expected a number") + 152 |
| c | error(2, '"Expected a number") + 8 | error(2, '"Syntax error") + 4 |
#+END:
#+BEGIN: aggregate :table "inputerrors" :cols "A (Q) (R)"
| A | (Q) | (R) |
|---+-------------------------------------+-------------------------------------------|
| a | [3, error(2, '"Expected a number")] | [10, 20] |
| b | [t, 1, 2] | [error(4, '"Expected a number"), 41, 111] |
| c | [8, error(2, '"Expected a number")] | [error(2, '"Syntax error"), 4] |
#+END:
* Test modifiers
Note the blank line between tblname and the actual table
#+tblname: bigprec
| A | Q | N |
|----+-------+---------------------|
| a | 12 | 20 |
| a | t+1 | 3.000000000000007 |
| bb | 77 | 4 |
| bb | 2*t | 5.12345678987654321 |
| bb | 2*t+1 | 6 |
#+BEGIN: aggregate :table "bigprec" :cols "A sum(Q) mean(Q);FS (Q)"
| A | sum(Q) | mean(Q) | (Q) |
|----+----------+--------------+--------------------|
| a | t + 13 | t / 2 + 13:2 | [12, t + 1] |
| bb | 4 t + 78 | 4:3 t + 26 | [77, 2 t, 2 t + 1] |
#+END:
#+BEGIN: aggregate :table "bigprec" :cols "A sum(N);p20f18 sum(N);%.5f mean(N);f15 (N);f3"
| A | sum(N) | sum(N) | mean(N) | (N) |
|----+-----------------------+----------+--------------------+---------------|
| a | 23.000000000000007000 | 23.00000 | 11.500000000000000 | [20, 3.000] |
| bb | 15.123456789876543210 | 15.12346 | 5.041152263290000 | [4, 5.123, 6] |
#+END:
* Test chaining
Result of an aggregation can be further processed, for example with another aggregation.
** chaining 3 aggregations
Note: header is 2 lines tall
#+TBLNAME: amx
| A | M | X |
| ~a | ~m | ~x |
|----+----+----|
| a | m | 1 |
| a | p | 2 |
| a | m | 3 |
|----+----+----|
| b | p | 4 |
| b | m | 5 |
| b | p | 6 |
| b | m | 7 |
#+TBLNAME: amsx
#+BEGIN: aggregate :table "amx" :cols "A M sum(X)"
| A | M | SX |
| ~a | ~m | ~x |
|----+----+----|
| a | m | 4 |
| a | p | 2 |
| b | p | 10 |
| b | m | 12 |
#+TBLFM: @1$3=SX
#+END:
#+TBLNAME: asx
#+BEGIN: aggregate :table "amsx" :cols "A sum(SX)"
#+caption: named_table
| A | SSX |
| ~a | ~x |
|----+-----|
| a | 6 |
| b | 22 |
#+TBLFM: @1$2=SSX
#+END:
#+BEGIN: aggregate :table "asx" :cols "sum(SSX)"
| sum(SSX) |
| ~x |
|----------|
| 28 |
#+END:
** chaining 2 transpositions
#+TBLNAME: tamx
#+BEGIN: transpose :table "amx"
| A | ~a | | a | a | a | | b | b | b | b |
| M | ~m | | m | p | m | | p | m | p | m |
| X | ~x | | 1 | 2 | 3 | | 4 | 5 | 6 | 7 |
#+END:
#+BEGIN: transpose :table "tamx"
| A | M | X |
| ~a | ~m | ~x |
|----+----+----|
| a | m | 1 |
| a | p | 2 |
| a | m | 3 |
|----+----+----|
| b | p | 4 |
| b | m | 5 |
| b | p | 6 |
| b | m | 7 |
#+END:
The double transposition is identical to the original "amx" table,
including horizontal lines
* Test funny column names
Name of columns are not unnecessarily alphanumeric words.
They need to be single or double quoted in formulas.
In a :cond lisp formula, only double quotes work.
** Quoted names
#+NAME: funnynames
# some additional ignored directives
# and blank lines
| first column | observed;number | computed/expected |
|--------------+-----------------+-------------------|
| a/experiment | 2.3 | 2.4 |
| a/experiment | 15.4 | 12.1 |
| a/experiment | 8.2 | 6.9 |
| b/test | -98.7 | 0.0 |
| b/test | 4.5 | 3.4 |
| b/test | 2.2 | 2.9 |
| zero | 0 | 0 |
#+BEGIN: aggregate :table "funnynames" :cols "\"first column\" mean('observed;number');%.3f mean('computed/expected');%.4f" :cond (and (>= (string-to-number "observed;number") 0) (not (equal "first column" "zero")))
| first column | mean(observed;number) | mean(computed/expected) |
|--------------+-----------------------+-------------------------|
| a/experiment | 8.633 | 7.1333 |
| b/test | 3.350 | 3.1500 |
#+END:
#+BEGIN: aggregate :table "funnynames" :cols ("'first column'" "mean('observed;number');%.3f" "mean('computed/expected');%.4f") :cond "(and (>= (string-to-number \"observed;number\") 0) (not (equal \"first column\" \"zero\")))"
| first column | mean(observed;number) | mean(computed/expected) |
|--------------+-----------------------+-------------------------|
| a/experiment | 8.633 | 7.1333 |
| b/test | 3.350 | 3.1500 |
#+END:
#+BEGIN: transpose :table "funnynames" :cols ("first column" "computed/expected" "observed;number")
| first column | | a/experiment | a/experiment | a/experiment | b/test | b/test | b/test | zero |
| computed/expected | | 2.4 | 12.1 | 6.9 | 0.0 | 3.4 | 2.9 | 0 |
| observed;number | | 2.3 | 15.4 | 8.2 | -98.7 | 4.5 | 2.2 | 0 |
#+END:
#+BEGIN: transpose :table "funnynames" :cols "'first column' 'computed/expected' 'observed;number'"
| first column | | a/experiment | a/experiment | a/experiment | b/test | b/test | b/test | zero |
| computed/expected | | 2.4 | 12.1 | 6.9 | 0.0 | 3.4 | 2.9 | 0 |
| observed;number | | 2.3 | 15.4 | 8.2 | -98.7 | 4.5 | 2.2 | 0 |
#+END:
** Non alphanumeric names
Accepted column names which do not require quoting:
- ascii letters
- numbers
- underscore _, dollar $, dot .
- accented letters like à é
- greek letters like α, Ω
- northen letters like ø
- russian letters like й
- esperanto letters like ŭ
#+NAME: non_alphanum
| _key.$ | v_A$4lé.à.α | on.eüΩ.øйŭ | 3.14 |
|--------+-------------+------------+------|
| a | 2.2 | 1 | 10 |
| a | 4.9 | 1 | 11 |
| b | 7.7 | 1 | 12 |
| b | 2.8 | 0 | 13 |
| b | 9.3 | 0 | 14 |
| c | 6.5 | 0 | 15 |
| a | 8.4 | 0 | 16 |
| a | 1.9 | 0 | 17 |
| b | 5.6 | 0 | 18 |
| c | 7.2 | 0 | 19 |
#+BEGIN: aggregate :table "non_alphanum" :cols "_key.$ vsum(v_A$4lé.à.α) vsum(on.eüΩ.øйŭ*10) vlist(on.eüΩ.øйŭ) vmean(3.14*1000)"
| _key.$ | vsum(v_A$4lé.à.α) | vsum(on.eüΩ.øйŭ*10) | vlist(on.eüΩ.øйŭ) | vmean(3.14*1000) |
|--------+-------------------+---------------------+-------------------+------------------|
| a | 17.4 | 20 | 1, 1, 0, 0 | 13500 |
| b | 25.4 | 10 | 1, 0, 0, 0 | 14250 |
| c | 13.7 | 0 | 0, 0 | 17000 |
#+END:
* Test malformed tables
Some columns are missing in some rows
This is on purpose
orgaggregate should tolerate such tables
Missing cells are handled as though they were empty
#+NAME: malformed
| Color | Level | Quantity | Day |
|-------+-------+----------+-----------|
| Red | 30 | 11 | Monday |
| Blue | 25 | 3 | Monday |
|
| Red | 45 | 15 | Tuesday |
| Blue | 33 | 18 | Tuesday |
| Red | 27 |
| Blue | 12 | 16 | Wednesday |
| Blue | 15 | 15 |
| Red | 39 | 24 | Thursday |
| Red | 41 | 29 | Thursday |
| Red | 49 | 30 | Thursday |
| Blue | 7 | 5 | Friday |
| Blue | 6 |
| Blue | 11 | 9 | Friday |
#+BEGIN: aggregate :table "malformed" :cols "Day count() sum(Quantity)"
| Day | count() | sum(Quantity) |
|-----------+---------+---------------|
| Monday | 2 | 14 |
| | 4 | 15 |
| Tuesday | 2 | 33 |
| Wednesday | 1 | 16 |
| Thursday | 3 | 83 |
| Friday | 2 | 14 |
#+END:
#+BEGIN: aggregate :table "malformed" :cols "Color vlist(l10)" :precompute "Level*10;'l10'"
| Color | vlist(l10) |
|-------+---------------------------------|
| Red | 300, 450, 270, 390, 410, 490 |
| Blue | 250, 330, 120, 150, 70, 60, 110 |
| | |
#+END:
* Test vlist($) vs. ($)
#+name: suitableforlist
| Day | Color | Level |
|-----------+------------+-------|
| Monday | Red | 20*30 |
| Monday | Blue | 55+25 |
| Tuesday | Red | 51 |
| Tuesday | Red | 45 |
| Tuesday | Blue | 33 |
| Wednesday | Red | 27 |
| Wednesday | Blue | 12 |
| Wednesday | Green | 15 |
| Thursday | Red | 39 |
| Thursday | Red | 41 |
| Thursday | Red+Green | 49 |
| Friday | Blue | (7) |
| Friday | Blue | (6+1) |
| Friday | Blue&Green | [11] |
#+BEGIN: aggregate :table "suitableforlist" :cols "Day vlist(Color) (Color) vlist(Level) (Level) Level*100 Level^2"
| Day | vlist(Color) | (Color) | vlist(Level) | (Level) | Level*100 | Level^2 |
|-----------+------------------------+-----------------------------------------+------------------+--------------+--------------------+---------|
| Monday | Red, Blue | [Red, Blue] | 20*30, 55+25 | [600, 80] | [60000, 8000] | 366400 |
| Tuesday | Red, Red, Blue | [Red, Red, Blue] | 51, 45, 33 | [51, 45, 33] | [5100, 4500, 3300] | 5715 |
| Wednesday | Red, Blue, Green | [Red, Blue, Green] | 27, 12, 15 | [27, 12, 15] | [2700, 1200, 1500] | 1098 |
| Thursday | Red, Red, Red+Green | [Red, Red, Red + Green] | 39, 41, 49 | [39, 41, 49] | [3900, 4100, 4900] | 5603 |
| Friday | Blue, Blue, Blue&Green | [Blue, Blue, error(4, '"Syntax error")] | (7), (6+1), [11] | [7, 7, [11]] | [700, 700, [1100]] | 219 |
#+END:
* Test sorting key alpha & numeric
#+NAME: unsortedtable
| p | q | x | Day | Color | Level | date |
|---+---+------+-----------+-------+-------+------------------------|
| 1 | b | 12.3 | Monday | Red | 2*30 | [2024-12-23 Mon 09:01] |
| 3 | b | 12.8 | Monday | Blue | 5+25 | [2019-11-24 Sun 13:04] |
| 1 | a | 3.5 | Tuesday | Red | 51 | [2029-09-25 Tue 13:54] |
| 2 | a | 3.9 | Tuesday | Red | 45 | [2033-09-26 Mon 03:55] |
| 3 | a | 3.5 | Tuesday | Blue | 33 | [2015-02-27 Fri 16:11] |
| 3 | a | 5.7 | Wednesday | Red | 97 | [2001-01-19 Fri 03:49] |
| 1 | a | 5.1 | Wednesday | Blue | 52 | [2035-12-26 Wed 00:00] |
|---+---+------+-----------+-------+-------+------------------------|
| 2 | b | 9.3 | Tuesday | Red | 39 | [2035-12-26 Wed 00:00] |
| 3 | b | 9.3 | Thursday | Red | 41 | [2002-01-19 Sat 23:22] |
| 4 | a | 1.4 | Friday | Blue | 79 | [2026-08-01 Sat 17:27] |
| 5 | a | 7.5 | Friday | Blue | 8+9 | [2020-09-15 Tue 13:07] |
| 4 | b | 8.2 | Thursday | Red | 41 | [2040-10-27 Sat 09:12] |
|---+---+------+-----------+-------+-------+------------------------|
| 5 | b | 1.1 | Wednesday | Red | 62 | [2011-01-29 Sat 15:06] |
#+BEGIN: aggregate :table "unsortedtable" :cols "p;^n Day;^a"
| p | Day |
|---+-----------|
| 1 | Monday |
| 1 | Tuesday |
| 1 | Wednesday |
| 2 | Tuesday |
| 3 | Monday |
| 3 | Thursday |
| 3 | Tuesday |
| 3 | Wednesday |
| 4 | Friday |
| 4 | Thursday |
| 5 | Friday |
| 5 | Wednesday |
#+END:
* Test sorting numeric expression
#+BEGIN: aggregate :table "unsortedtable" :cols "Day count();^N"
| Day | count() |
|-----------+---------|
| Tuesday | 4 |
| Wednesday | 3 |
| Monday | 2 |
| Thursday | 2 |
| Friday | 2 |
#+END:
#+BEGIN: aggregate :table "unsortedtable" :cols "Day vsum(Level);^n"
| Day | vsum(Level) |
|-----------+-------------|
| Thursday | 82 |
| Monday | 90 |
| Friday | 96 |
| Tuesday | 168 |
| Wednesday | 211 |
#+END:
* Test sorting hline
#+BEGIN: aggregate :table "unsortedtable" :cols "hline;^N q;^a count()"
| hline | q | count() |
|-------+---+---------|
| 2 | b | 1 |
| 1 | a | 2 |
| 1 | b | 3 |
| 0 | a | 5 |
| 0 | b | 2 |
#+END:
* Test sorting dates-times
#+BEGIN: aggregate :table "unsortedtable" :cols "date;^T count()"
| date | count() |
|------------------------+---------|
| [2040-10-27 Sat 09:12] | 1 |
| [2035-12-26 Wed 00:00] | 2 |
| [2033-09-26 Mon 03:55] | 1 |
| [2029-09-25 Tue 13:54] | 1 |
| [2026-08-01 Sat 17:27] | 1 |
| [2024-12-23 Mon 09:01] | 1 |
| [2020-09-15 Tue 13:07] | 1 |
| [2019-11-24 Sun 13:04] | 1 |
| [2015-02-27 Fri 16:11] | 1 |
| [2011-01-29 Sat 15:06] | 1 |
| [2002-01-19 Sat 23:22] | 1 |
| [2001-01-19 Fri 03:49] | 1 |
#+END:
* Test sorting major-minor columns
#+BEGIN: aggregate :table "unsortedtable" :cols "date;^t3 Color;^a2 x;^n1"
| date | Color | x |
|------------------------+-------+------|
| [2011-01-29 Sat 15:06] | Red | 1.1 |
| [2026-08-01 Sat 17:27] | Blue | 1.4 |
| [2015-02-27 Fri 16:11] | Blue | 3.5 |
| [2029-09-25 Tue 13:54] | Red | 3.5 |
| [2033-09-26 Mon 03:55] | Red | 3.9 |
| [2035-12-26 Wed 00:00] | Blue | 5.1 |
| [2001-01-19 Fri 03:49] | Red | 5.7 |
| [2020-09-15 Tue 13:07] | Blue | 7.5 |
| [2040-10-27 Sat 09:12] | Red | 8.2 |
| [2002-01-19 Sat 23:22] | Red | 9.3 |
| [2035-12-26 Wed 00:00] | Red | 9.3 |
| [2024-12-23 Mon 09:01] | Red | 12.3 |
| [2019-11-24 Sun 13:04] | Blue | 12.8 |
#+END:
* Test sorting push
#+ORGTBL: SEND sortag1 orgtbl-to-aggregated-table :cols "cölØr vsum(vâluε);^N count();^N vmean('ra;han');f3"
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
| Red | 1.1 | 58 |
#+BEGIN RECEIVE ORGTBL sortag1
| cölØr | vsum(vâluε) | count() | vmean(ra;han) |
|--------+-------------+---------+---------------|
| Yellow | 23.5 | 4 | 50.250 |
| Blue | 20.6 | 3 | 48.333 |
| Red | 12.4 | 5 | 53.800 |
#+END RECEIVE ORGTBL sortag1
* Test hline output
#+name: withhline
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
|--------+-------+--------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
|--------+-------+--------|
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+--------|
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
Are original hlines given back?
#+BEGIN: aggregate :table "withhline" :cols "cölØr vâluε 'ra;han'" :hline 1
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
|--------+-------+--------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
|--------+-------+--------|
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
|--------+-------+--------|
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
#+END:
I do not specify hlines in the output
#+BEGIN: aggregate :table "withhline" :cols "cölØr vâluε 'ra;han'"
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Yellow | 9.1 | 95 |
| Red | 2.6 | 84 |
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Yellow | 5.4 | 17 |
| Blue | 4.9 | 64 |
| Red | 3.9 | 51 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
| Red | 1.1 | 58 |
| Yellow | 3.4 | 51 |
#+END:
What if I want hline on cölØr?
#+BEGIN: aggregate :table "withhline" :cols "cölØr;^a vâluε 'ra;han'" :hline 1
| cölØr | vâluε | ra;han |
|--------+-------+--------|
| Blue | 8.7 | 52 |
| Blue | 7.0 | 29 |
| Blue | 4.9 | 64 |
|--------+-------+--------|
| Red | 1.3 | 41 |
| Red | 3.5 | 35 |
| Red | 2.6 | 84 |
| Red | 3.9 | 51 |
| Red | 1.1 | 58 |
|--------+-------+--------|
| Yellow | 9.1 | 95 |
| Yellow | 5.4 | 17 |
| Yellow | 2.4 | 55 |
| Yellow | 6.6 | 34 |
| Yellow | 3.4 | 51 |
#+END:
And if I explicitly require hline column?
#+BEGIN: aggregate :table "withhline" :cols "hline;^n cölØr;^a vâluε 'ra;han'"
| hline | cölØr | vâluε | ra;han |
|-------+--------+-------+--------|
| 0 | Red | 1.3 | 41 |
| 0 | Red | 3.5 | 35 |
| 0 | Red | 2.6 | 84 |
| 0 | Yellow | 9.1 | 95 |
| 1 | Blue | 8.7 | 52 |
| 1 | Blue | 7.0 | 29 |
| 1 | Yellow | 5.4 | 17 |
| 2 | Blue | 4.9 | 64 |
| 2 | Red | 3.9 | 51 |
| 2 | Yellow | 2.4 | 55 |
| 2 | Yellow | 6.6 | 34 |
| 3 | Red | 1.1 | 58 |
| 3 | Yellow | 3.4 | 51 |
#+END:
And hline rows as well as column?
#+BEGIN: aggregate :table "withhline" :cols "hline;^N cölØr;^a vâluε 'ra;han'" :hline 1
| hline | cölØr | vâluε | ra;han |
|-------+--------+-------+--------|
| 3 | Red | 1.1 | 58 |
| 3 | Yellow | 3.4 | 51 |
|-------+--------+-------+--------|
| 2 | Blue | 4.9 | 64 |
| 2 | Red | 3.9 | 51 |
| 2 | Yellow | 2.4 | 55 |
| 2 | Yellow | 6.6 | 34 |
|-------+--------+-------+--------|
| 1 | Blue | 8.7 | 52 |
| 1 | Blue | 7.0 | 29 |
| 1 | Yellow | 5.4 | 17 |
|-------+--------+-------+--------|
| 0 | Red | 1.3 | 41 |
| 0 | Red | 3.5 | 35 |
| 0 | Red | 2.6 | 84 |
| 0 | Yellow | 9.1 | 95 |
#+END:
Same with hline & cölØr to separate blocks
#+BEGIN: aggregate :table "withhline" :cols "hline;^N cölØr;^a vâluε 'ra;han'" :hline 2
| hline | cölØr | vâluε | ra;han |
|-------+--------+-------+--------|
| 3 | Red | 1.1 | 58 |
|-------+--------+-------+--------|
| 3 | Yellow | 3.4 | 51 |
|-------+--------+-------+--------|
| 2 | Blue | 4.9 | 64 |
|-------+--------+-------+--------|
| 2 | Red | 3.9 | 51 |
|-------+--------+-------+--------|
| 2 | Yellow | 2.4 | 55 |
| 2 | Yellow | 6.6 | 34 |
|-------+--------+-------+--------|
| 1 | Blue | 8.7 | 52 |
| 1 | Blue | 7.0 | 29 |
|-------+--------+-------+--------|
| 1 | Yellow | 5.4 | 17 |
|-------+--------+-------+--------|
| 0 | Red | 1.3 | 41 |
| 0 | Red | 3.5 | 35 |
| 0 | Red | 2.6 | 84 |
|-------+--------+-------+--------|
| 0 | Yellow | 9.1 | 95 |
#+END:
* Test filter only
#+name: planet
| planet | mass kg | dist MKM |
|---------+-----------+----------|
| Sun | 1.9891e30 | 0 |
| Mercury | 3.3022e23 | 60 |
| Venus | 4.8685e24 | 100 |
| Earth | 5.9736e24 | 150 |
| Mars | 6.4185e23 | 220 |
| Jupiter | 1.8986e27 | 780 |
| Saturn | 5.6846e26 | 1420 |
| Uranus | 8.6810e25 | 2870 |
| Neptune | 10.243e25 | 4500 |
| Pluto | 1.25e22 | 5800 |
Without :cols parameter, we get all columns
#+BEGIN: aggregate :table "planet" :cond (> (string-to-number "dist MKM") 150)
| planet | mass kg | dist MKM |
|---------+-----------+----------|
| Mars | 6.4185e23 | 220 |
| Jupiter | 1.8986e27 | 780 |
| Saturn | 5.6846e26 | 1420 |
| Uranus | 8.6810e25 | 2870 |
| Neptune | 10.243e25 | 4500 |
| Pluto | 1.25e22 | 5800 |
#+END:
What happens without column names in the input?
#+name: planetnh
| Sun | 1.9891e30 | 0 |
| Mercury | 3.3022e23 | 60 |
| Venus | 4.8685e24 | 100 |
| Earth | 5.9736e24 | 150 |
| Mars | 6.4185e23 | 220 |
| Jupiter | 1.8986e27 | 780 |
| Saturn | 5.6846e26 | 1420 |
| Uranus | 8.6810e25 | 2870 |
| Neptune | 10.243e25 | 4500 |
| Pluto | 1.25e22 | 5800 |
#+BEGIN: aggregate :table "planetnh" :cond (<= (string-to-number "$3") 150)
| $1 | $2 | $3 |
|---------+-----------+-----|
| Sun | 1.9891e30 | 0 |
| Mercury | 3.3022e23 | 60 |
| Venus | 4.8685e24 | 100 |
| Earth | 5.9736e24 | 150 |
#+END:
* Test custom column names
#+BEGIN: aggregate :table "pulledtable" :cols "pé;^n vsum(xà);'sum_of_xà' vmean(yÿ);'average Ÿ' vmax(zö);'MAX of ZÖ'"
| pé | sum_of_xà | average Ÿ | MAX of ZÖ |
|----+-----------+---------------+-----------|
| 1 | 20.1 | 4 | 9 |
| 2 | 12 | 5 | 5 |
| 3 | 29 | 5.5 | 9 |
| 4 | a + b | a / 2 + b / 2 | 8 |
| 5 | a + b + 3 | 5 a + b^2 / 2 | 8 |
#+END:
* Test no collision
There should be no collision between column names and reserved Calc function names.
For instance ~vsum~, which is a Calc function, should be usable as a column name.
#+name: keyword-collision
| vmean | sort | vsum | sum | vmax | aaa |
|-------+------+------+-----+------+-----|
| 2 | 12.3 | 43 | 43 | 1 | 8.2 |
| 8 | 34.4 | 81 | 81 | 1 | 9.3 |
| 4 | 51.5 | 40 | 40 | 1 | 1.3 |
| 5 | 8.1 | 27 | 27 | 2 | 3.9 |
| 2 | 4.7 | 41 | 41 | 2 | 3.5 |
| 9 | 33.9 | 62 | 62 | 3 | 2.1 |
| 1 | 41.7 | 83 | 83 | 3 | 2.7 |
#+BEGIN: aggregate :table "keyword-collision" :cols "vmax count() vsum(vmean) vsum(sort) sort(vsum) sort(sum) vmean(sum);%.2f vmean(vsum);f2"
| vmax | count() | vsum(vmean) | vsum(sort) | sort(vsum) | sort(sum) | vmean(sum) | vmean(vsum) |
|------+---------+-------------+------------+--------------+--------------+------------+-------------|
| 1 | 3 | 14 | 98.2 | [40, 43, 81] | [40, 43, 81] | 54.67 | 54.67 |
| 2 | 2 | 7 | 12.8 | [27, 41] | [27, 41] | 34.00 | 34 |
| 3 | 2 | 10 | 75.6 | [62, 83] | [62, 83] | 72.50 | 72.50 |
#+END:
* Test disordered formatters & decorators
#+BEGIN: aggregate :table "planet" :cols "planet vmax('mass kg');^n;e4;'MassKG' vmin('dist MKM')*1e6;^N;'DistKM';e2"
| planet | MassKG | DistKM |
|---------+----------+--------|
| Pluto | 12.5e21 | 5.8e9 |
| Mercury | 330.2e21 | 60e6 |
| Mars | 641.9e21 | 220e6 |
| Venus | 4.869e24 | 100e6 |
| Earth | 5.974e24 | 150e6 |
| Uranus | 86.81e24 | 2.9e9 |
| Neptune | 102.4e24 | 4.5e9 |
| Saturn | 568.5e24 | 1.4e9 |
| Jupiter | 1.899e27 | 780e6 |
| Sun | 1.989e30 | 0e0 |
#+END:
* Test lambda post-processing
#+BEGIN: aggregate :table "pulledtable" :cols "qû vsum(zö)" :post (lambda (table) (append table '(hline (c 112233))))
| qû | vsum(zö) |
|----+----------|
| b | 38 |
| a | 29 |
|----+----------|
| c | 112233 |
#+END:
#+BEGIN: aggregate :table "pulledtable" :cols "qû vsum(zö)" :post (lambda (table) (append '((c 112233) hline) table))
| c | 112233 |
|----+----------|
| qû | vsum(zö) |
|----+----------|
| b | 38 |
| a | 29 |
#+END:
* Test babel post-processing
#+BEGIN: aggregate :table "pulledtable" :cols "qû vsum(zö)" :post "post-proc-babel(*this*)"
| AA | BB |
|----+--------|
| c | 112233 |
|----+--------|
| b | 38 |
| a | 29 |
#+END:
#+name: post-proc-babel
#+begin_src elisp :var intbl="" :colnames '(AA BB)
(append
'((c 112233) hline)
intbl))
#+end_src
* Test push lambda post-processing
#+ORGTBL: SEND sent-aggregate-post orgtbl-to-aggregated-table :cols "a vsum(b) vsum(c)" :post (lambda (tbl) (append tbl '(hline (h 9 "8"))))
#+ORGTBL: SEND sent-transpose-post orgtbl-to-transposed-table :cols "a c" :post (lambda (tbl) (append tbl '(hline (h "" 3.4 "8.8"))))
| a | b | c |
|---+----+----|
| x | 34 | 56 |
| i | 90 | 12 |
| x | 51 | 3 |
| i | 1 | 11 |
#+BEGIN RECEIVE ORGTBL sent-aggregate-post
| a | vsum(b) | vsum(c) |
|---+---------+---------|
| x | 85 | 59 |
| i | 91 | 23 |
|---+---------+---------|
| h | 9 | 8 |
#+END RECEIVE ORGTBL sent-aggregate-post
#+BEGIN RECEIVE ORGTBL sent-transpose-post
| a | | x | i | x | i |
| c | | 56 | 12 | 3 | 11 |
|---+---+-----+-----+---+----|
| h | | 3.4 | 8.8 |
#+END RECEIVE ORGTBL sent-transpose-post
* Test push babel post-processing
#+ORGTBL: SEND sent-transpose-post-babel orgtbl-to-transposed-table :cols "p r q" :post "post-proc-babel-send(*this*)"
#+ORGTBL: SEND sent-aggregate-post-babel orgtbl-to-aggregated-table :cols "p vsum(q) vsum(r)" :post "post-proc-babel-send(intbl=*this*)"
| q | p | r |
|-------+---+-------|
| 34.9 | x | 56.1 |
| 9.20 | i | 77.2 |
| 51.29 | x | 3.86 |
| 76.7 | i | 19.47 |
#+BEGIN RECEIVE ORGTBL sent-aggregate-post-babel
| x | 86.19 | 59.96 |
| i | 85.9 | 96.67 |
| add | 3.1416 | 5.6 |
|-----+--------+-------|
| sub | 2.718 | -42.0 |
#+END RECEIVE ORGTBL sent-aggregate-post-babel
#+BEGIN RECEIVE ORGTBL sent-transpose-post-babel
| p | | x | i | x | i |
| r | | 56.1 | 77.2 | 3.86 | 19.47 |
| q | | 34.9 | 9.20 | 51.29 | 76.7 |
| add | 3.1416 | 5.6 |
|-----+--------+-------+------+-------+-------|
| sub | 2.718 | -42.0 |
#+END RECEIVE ORGTBL sent-transpose-post-babel
#+name: post-proc-babel-send
#+begin_src elisp :var intbl=""
(append
intbl
'((add 3.1416 "5.6") hline (sub 2.718 "-42.0")))
#+end_src
* Japanese characters
Japanese characters are wider than ASCII ones.
In mono-spaced fonts, they are often 2 times wider.
Not all fonts are equal. The Ubuntu one is not too bad, although not perfect:
: (set-face-font 'default "Ubuntu Mono")
#+name: 日本のテーブル
| 如何 | 量 |
|--------------+----|
| 急行電車 | 23 |
| 山に雪が降る | 21 |
| 鳥と花 | 34 |
| 急行電車 | 61 |
| 鳥と花 | 93 |
| 山に雪が降る | 48 |
#+BEGIN: aggregate :table "日本のテーブル" :cols "如何 vsum(量)"
| 如何 | vsum(量) |
|--------------+----------|
| 急行電車 | 84 |
| 山に雪が降る | 69 |
| 鳥と花 | 127 |
#+END:
* Alignment cookies
What to do with cookies?
<> <12>
They are not real data, rather metadata. Mixed into data, they may
result in false aggregations. Therefore they should be ignored.
But in the header of tables, cookies do not change aggregated
results. They format the source column. Probably the aggregated column
may benefit from the same formatting. Therefore, cookies are kept in
headers.
#+name: with-cookies
| color | quantity | level |
| | | <3> |
| kolor | kiom | nivelo |
|--------+----------+--------|
| yellow | 72 | 3 |
| green | 55 | 0 |
| | | 4 |
| orange | 80 | 0 |
| white | 19 | 6 |
| green | 4 | 4 |
| yellow | 58 | 5 |
| | <25> | 0 |
| orange | 22 | 4 |
| orange | 7 | 4 |
| <> | <> | 2 |
| red | 71 | 3 |
| blue | 56 | 3 |
| red | 52 | 5 |
| <7> | | 3 |
| orange | 35 | 0 |
| | | 3 |
| yellow | 23 | 0 |
| | <44> | 0 |
| blue | 93 | 4 |
| black | | 0 |
| green | 82 | 2 |
| <9> | <4> | 5 |
#+BEGIN: aggregate :table "with-cookies" :cols "color vsum(quantity);'sum' count();'nb' vsum(quantity)/vmean(level);'leveled'"
| color | sum | nb | leveled |
| | | | |
| kolor | kiom | | |
|--------+------+----+---------------|
| yellow | 153 | 3 | 57.3749999999 |
| green | 141 | 3 | 70.5 |
| orange | 144 | 4 | 72 |
| white | 19 | 1 | 3.16666666667 |
| red | 123 | 2 | 30.75 |
| blue | 149 | 2 | 42.5714285714 |
#+END:
#+BEGIN: transpose :table "with-cookies"
| color | | kolor | | yellow | green | orange | white | green | yellow | orange | orange | red | blue | red | orange | yellow | blue | green |
| quantity | | kiom | | 72 | 55 | 80 | 19 | 4 | 58 | 22 | 7 | 71 | 56 | 52 | 35 | 23 | 93 | 82 |
| level | <3> | nivelo | | 3 | 0 | 0 | 6 | 4 | 5 | 4 | 4 | 3 | 3 | 5 | 0 | 0 | 4 | 2 |
#+END:
* 1st data row is not the header
When the input table does not have a header,
the first data row should not be mistaken with a header.
#+name: missing-header
| a | 12 | 33 |
| c | 13 | 12 |
| x | 14 | 12 |
| y | 15 | 45 |
| z | 7 | 7 |
#+BEGIN: aggregate :table "missing-header" :cols "$1" :cond (equal $3 "12")
| $1 |
|----|
| c |
| x |
#+END:
In case of a mistake, the result is:
| $1 |
|----|
| z |
* Debug settings
4 settings: c q C Q
#+BEGIN: aggregate :table "withhline" :cols "hline vsum(vâluε);c vsum(vâluε);q vsum(vâluε);C vsum(vâluε);Q"
| hline | vsum(vâluε) | vsum(vâluε) | vsum(vâluε) | vsum(vâluε) |
|-------+-------------+-----------------------------------+----------------------------+-------------------------------------------------------------------------------|
| 0 | vsum($2) | (calcFunc-vsum (calcFunc-Frux 2)) | vsum([1.3, 3.5, 9.1, 2.6]) | (calcFunc-vsum (vec (float 13 -1) (float 35 -1) (float 91 -1) (float 26 -1))) |
| 1 | vsum($2) | (calcFunc-vsum (calcFunc-Frux 2)) | vsum([8.7, 7., 5.4]) | (calcFunc-vsum (vec (float 87 -1) (float 7 0) (float 54 -1))) |
| 2 | vsum($2) | (calcFunc-vsum (calcFunc-Frux 2)) | vsum([4.9, 3.9, 2.4, 6.6]) | (calcFunc-vsum (vec (float 49 -1) (float 39 -1) (float 24 -1) (float 66 -1))) |
| 3 | vsum($2) | (calcFunc-vsum (calcFunc-Frux 2)) | vsum([1.1, 3.4]) | (calcFunc-vsum (vec (float 11 -1) (float 34 -1))) |
#+END:
* hline as a column to be aggregated
Does it make sense to calculate something based on hline?
Anyway, it is now available at no cost.
#+BEGIN: aggregate :table "withhline" :cols "cölØr vmean(hline*15) vlist(hline)"
| cölØr | vmean(hline*15) | vlist(hline) |
|--------+-----------------+---------------|
| Red | 15 | 0, 0, 0, 2, 3 |
| Yellow | 24 | 0, 1, 2, 2, 3 |
| Blue | 20 | 1, 1, 2 |
#+END:
We see that Reds are more at the begining of the input table,
while Yellows are more at the end.
* Input table is a Babel script
Note the =:colnames yes= parameter to output a header with label & value
column names.
The table resulting from the =ascript= script is computed on the fly, it
appears nowhere in the buffer.
#+name: ascript
#+begin_src elisp :colnames yes
`(
(label "value") ; cells are symbols or strings
hline
,@(cl-loop
for i from 1 to 20
collect
(list
(format "%c" (+ ?a (% i 3))) ; cell is a string
i))) ; cell is a number
#+end_src
Use column names in the =:cols= specification:
#+BEGIN: aggregate :table "ascript" :cols "label vsum(value)"
| label | vsum(value) |
|-------+-------------|
| b | 70 |
| c | 77 |
| a | 63 |
#+END:
Use dollar to specify columns in =:cols=:
#+BEGIN: aggregate :table "ascript" :cols "$1 vsum($2)"
| $1 | vsum($2) |
|----+----------|
| b | 70 |
| c | 77 |
| a | 63 |
#+END:
a script with a parameter:
#+name: ascriptparam
#+begin_src elisp :colnames yes :var len=20
`(
("label" value) ; cells are symbols or strings
hline
,@(cl-loop
for i from 1 to len
collect
(list
(format "%c" (+ ?a (% i 3))) ; cell is a string
i))) ; cell is a number
#+end_src
#+BEGIN: aggregate :table "ascriptparam(len=10)" :cols "label vsum(value)"
| label | vsum(value) |
|-------+-------------|
| b | 22 |
| c | 15 |
| a | 18 |
#+END:
a longer table is generated (100 rows),
but only 10 rows are retained (12 = 10 rows + header + hline)
#+BEGIN: aggregate :table "ascriptparam(len=100)[0:12]" :cols "label vsum(value)"
| label | vsum(value) |
|-------+-------------|
| b | 22 |
| c | 26 |
| a | 18 |
#+END:
* Column & single-cell formulas
Is the Org Mode bug overcome?
It happens when
- there are both a column formula and a single cell formula
- they need to add new columns
#+BEGIN: aggregate :table "planet" :cols "planet vsum('dist MKM');'km'"
| planet | km | au |
|---------+------+-------|
| Sun | 0 | 0.00 |
| Mercury | 60 | 0.40 |
| Venus | 100 | 0.67 |
| Earth | 150 | 1.00 |
| Mars | 220 | 1.47 |
| Jupiter | 780 | 5.20 |
| Saturn | 1420 | 9.47 |
| Uranus | 2870 | 19.13 |
| Neptune | 4500 | 30.00 |
| Pluto | 5800 | 38.67 |
#+TBLFM: $3=$2/150;%.2f::@1$3=au
#+END:
* Aggregate on computed bins
Sometimes, input columns are not enough to aggregate on.
Virtual computed columns may be handy.
#+name: want-month
| Date | Quty |
|------------------+------|
| [2027-02-10 mer] | 38 |
| [2027-02-21 dim] | 58 |
| [2027-03-04 ĵaŭ] | 52 |
| [2027-03-15 lun] | 35 |
| [2027-03-26 ven] | 62 |
| [2027-04-06 mar] | 19 |
| [2027-04-17 sab] | 22 |
| [2027-04-28 mer] | 63 |
| [2027-05-09 dim] | 70 |
| [2027-05-20 ĵaŭ] | 51 |
| [2027-05-31 lun] | 55 |
| [2027-06-11 ven] | 49 |
| [2027-06-22 mar] | 96 |
| [2027-07-03 sab] | 62 |
| [2027-07-14 mer] | 7 |
| [2027-07-25 dim] | 43 |
Aggregate on months extracted from 'Date':
#+BEGIN: aggregate :table "want-month" :cols "Month vsum(Quty)" :precompute ("month(Date);'Month'")
| Month | vsum(Quty) |
|-------+------------|
| 2 | 96 |
| 3 | 149 |
| 4 | 104 |
| 5 | 176 |
| 6 | 145 |
| 7 | 112 |
#+END:
An input table with the first column acting as a tag, and the second as a quantity:
#+name: want-bins
| 129 | 32.56 |
| 133 | 71.45 |
| 139 | 72.80 |
| 172 | 14.99 |
| 343 | 88.58 |
| 373 | 51.56 |
| 406 | 87.66 |
| 444 | 14.13 |
| 459 | 52.87 |
| 510 | 59.10 |
| 527 | 78.41 |
| 634 | 23.71 |
| 673 | 91.14 |
| 739 | 83.03 |
| 750 | 28.13 |
| 757 | 60.63 |
| 792 | 64.62 |
| 833 | 13.28 |
| 848 | 29.89 |
| 871 | 82.70 |
| 945 | 22.95 |
| 967 | 42.58 |
We want to aggregate on coarse bins made of hundredths of the firt column:
#+BEGIN: aggregate :table "want-bins" :cols "$3 vmean($2);f2" :precompute ("'(floor (/ (string-to-number $1) 100))")
| $3 | vmean($2) |
|----+-----------|
| 1 | 47.95 |
| 3 | 70.07 |
| 4 | 51.55 |
| 5 | 68.76 |
| 6 | 57.43 |
| 7 | 59.10 |
| 8 | 41.96 |
| 9 | 32.77 |
#+END:
Same aggregation, with additional metrics (min & max):
#+BEGIN: aggregate :table "want-bins" :cols "$3 vmin(H) vmax(H) vmean($2);f2" :precompute "'(floor (/ (string-to-number $1) 100)) :: $1/100;'H'"
| $3 | vmin(H) | vmax(H) | vmean($2) |
|----+---------+---------+-----------|
| 1 | 1.29 | 1.72 | 47.95 |
| 3 | 3.43 | 3.73 | 70.07 |
| 4 | 4.06 | 4.59 | 51.55 |
| 5 | 5.1 | 5.27 | 68.76 |
| 6 | 6.34 | 6.73 | 57.43 |
| 7 | 7.39 | 7.92 | 59.10 |
| 8 | 8.33 | 8.71 | 41.96 |
| 9 | 9.45 | 9.67 | 32.77 |
#+END:
Let us format a pre-computed column:
#+BEGIN: aggregate :table "want-bins" :cols "LL list(H)" :precompute "'(floor (/ (string-to-number $1) 100));%.3f;'LL' :: $1/100;'H';%.1f"
| LL | list(H) |
|-------+--------------------|
| 1.000 | 1.3, 1.3, 1.4, 1.7 |
| 3.000 | 3.4, 3.7 |
| 4.000 | 4.1, 4.4, 4.6 |
| 5.000 | 5.1, 5.3 |
| 6.000 | 6.3, 6.7 |
| 7.000 | 7.4, 7.5, 7.6, 7.9 |
| 8.000 | 8.3, 8.5, 8.7 |
| 9.000 | 9.4, 9.7 |
#+END:
* Distant tables
table by name
#+BEGIN: aggregate :table "distant-tests.org:distanttable" :cols "tag vsum(val)"
| tag | vsum(val) |
|-----+-----------|
| A | 405.43 |
| BB | 453.63 |
| CCC | 215.71 |
#+END:
Babel by name
#+BEGIN: aggregate :table "distant-tests.org:distantbabel" :cols "tag vsum(value)"
| tag | vsum(value) |
|-----+-------------|
| BB | 825 |
| CCC | 818 |
| A | 832 |
#+END:
Babel by name and parameter
#+BEGIN: aggregate :table "distant-tests.org:distantbabel(factor=17)" :cols "tag vsum(value)"
| tag | vsum(value) |
|-----+-------------|
| BB | 825 |
| CCC | 1114 |
| A | 536 |
#+END:
Babel, mean of floating points
#+BEGIN: aggregate :table "distant-tests.org:distantbabel(factor=17)" :cols "tag vmean(inv) count()"
| tag | vmean(inv) | count() |
|-----+----------------+---------|
| BB | 0.976763739953 | 17 |
| CCC | 0.975810407486 | 17 |
| A | 0.976263808395 | 16 |
#+END:
table by ID
#+BEGIN: aggregate :table "55ab27a2-c44b-4a14-9ba4-f6879375207d[0:7]" :cols "ref vsum(val)"
| ref | vsum(val) |
|------+-----------|
| S | 685.6 |
| TT | 996.4 |
| UUU | 945.2 |
| VVVV | 974.7 |
#+END:
* CSV table
abundance of large cities per country
the source CSV file is TAB-separated, which is guessed
#+BEGIN: aggregate :table "geography-a.csv:(csv)" :cols "$2 count();^N vsum($3)"
| $2 | count() | vsum($3) |
|----------------+---------+-----------|
| China | 28 | 264183191 |
| India | 10 | 140092165 |
| Japan | 4 | 71117892 |
| Brazil | 4 | 48284586 |
| Mexico | 3 | 33697146 |
| Bangladesh | 2 | 30197563 |
| Egypt | 2 | 28897566 |
| Pakistan | 2 | 32895483 |
| Nigeria | 2 | 21763797 |
| Turkey | 2 | 21777496 |
| Russia | 2 | 18346030 |
| Vietnam | 2 | 15392473 |
| Saudi Arabia | 2 | 12980333 |
| Spain | 2 | 12577695 |
| South Africa | 2 | 11501203 |
| Australia | 2 | 10663074 |
| DR Congo | 1 | 17815364 |
| Argentina | 1 | 15714124 |
| Philippines | 1 | 15211511 |
| Colombia | 1 | 11779275 |
| Indonesia | 1 | 11628728 |
| Peru | 1 | 11529982 |
| Thailand | 1 | 11415533 |
| France | 1 | 11352823 |
| Angola | 1 | 10049628 |
| South Korea | 1 | 10059272 |
| United Kingdom | 1 | 9818142 |
| Iran | 1 | 9738111 |
| Malaysia | 1 | 8980578 |
| Tanzania | 1 | 8529744 |
| Iraq | 1 | 8154140 |
| United States | 1 | 7966324 |
| Hong Kong | 1 | 7791531 |
| Chile | 1 | 6973392 |
| Sudan | 1 | 6778168 |
| Canada | 1 | 6513813 |
| Singapore | 1 | 6167759 |
| Ivory Coast | 1 | 6054358 |
| Ethiopia | 1 | 5961711 |
| Myanmar | 1 | 5829964 |
| Kenya | 1 | 5772121 |
| Afghanistan | 1 | 4862586 |
| Cameroon | 1 | 4859198 |
| Israel | 1 | 4577871 |
| Taiwan | 1 | 4570576 |
#+END:
A CVS table oddly formatted, with a header:
#+BEGIN: aggregate :table "hline.csv:(csv header)" :cols "cc vsum(bb) vmean(aa)"
| cc | vsum(bb) | vmean(aa) |
|-------+----------+-----------|
| apple | 831 | 62.2 |
| grape | -22 | 13.75 |
#+END:
Here we add an external header, even though there is already a header
in the CSV file, and see what happens:
#+BEGIN: aggregate :table "hline.csv:(csv colnames (level quantity fruit))" :cols "fruit vsum(quantity)"
| fruit | vsum(quantity) |
|-------+----------------|
| cc | bb |
| apple | 831 |
| grape | -22 |
#+END:
Check if horizontal separators are recognized:
#+BEGIN: aggregate :table "hline.csv:(csv header)" :cols "hline cc vsum(bb)" :hline 1
| hline | cc | vsum(bb) |
|-------+-------+----------|
| 0 | apple | 654 |
| 0 | grape | -78 |
|-------+-------+----------|
| 1 | grape | 45 |
| 1 | apple | 192 |
|-------+-------+----------|
| 2 | apple | -15 |
| 2 | grape | 11 |
#+END:
CSV in distant Org file
#+BEGIN: aggregate :table "distant-tests.org:distantcsv(csv header)" :cols "label vsum(quantity)"
| label | vsum(quantity) |
|-------+----------------|
| jes | 70 |
| ne | -106 |
#+END:
* JSON table
same as in CSV
#+BEGIN: aggregate :table "geography-a.json:(json)" :cols "$2 count();^N vsum($3)"
| $2 | count() | vsum($3) |
|----------------+---------+-----------|
| China | 28 | 264183191 |
| India | 10 | 140092165 |
| Japan | 4 | 71117892 |
| Brazil | 4 | 48284586 |
| Mexico | 3 | 33697146 |
| Bangladesh | 2 | 30197563 |
| Egypt | 2 | 28897566 |
| Pakistan | 2 | 32895483 |
| Nigeria | 2 | 21763797 |
| Turkey | 2 | 21777496 |
| Russia | 2 | 18346030 |
| Vietnam | 2 | 15392473 |
| Saudi Arabia | 2 | 12980333 |
| Spain | 2 | 12577695 |
| South Africa | 2 | 11501203 |
| Australia | 2 | 10663074 |
| DR Congo | 1 | 17815364 |
| Argentina | 1 | 15714124 |
| Philippines | 1 | 15211511 |
| Colombia | 1 | 11779275 |
| Indonesia | 1 | 11628728 |
| Peru | 1 | 11529982 |
| Thailand | 1 | 11415533 |
| France | 1 | 11352823 |
| Angola | 1 | 10049628 |
| South Korea | 1 | 10059272 |
| United Kingdom | 1 | 9818142 |
| Iran | 1 | 9738111 |
| Malaysia | 1 | 8980578 |
| Tanzania | 1 | 8529744 |
| Iraq | 1 | 8154140 |
| United States | 1 | 7966324 |
| Hong Kong | 1 | 7791531 |
| Chile | 1 | 6973392 |
| Sudan | 1 | 6778168 |
| Canada | 1 | 6513813 |
| Singapore | 1 | 6167759 |
| Ivory Coast | 1 | 6054358 |
| Ethiopia | 1 | 5961711 |
| Myanmar | 1 | 5829964 |
| Kenya | 1 | 5772121 |
| Afghanistan | 1 | 4862586 |
| Cameroon | 1 | 4859198 |
| Israel | 1 | 4577871 |
| Taiwan | 1 | 4570576 |
#+END:
Same as CSV, vector of vectors, first vector is the header
#+BEGIN: aggregate :table "hline-header.json:(json header)" :cols "cc vsum(bb) vmean(aa)"
| cc | vsum(bb) | vmean(aa) |
|-------+----------+-----------|
| apple | 831 | 62.2 |
| grape | -22 | 13.75 |
#+END:
Same test, input is mixed vectors and hashed-objects,
resulting header is a mixture of first vector and keys of hashed-objects.
#+BEGIN: aggregate :table "hline-hash.json:(json header)" :cols "cc vsum(bb) vmean(aa)"
| cc | vsum(bb) | vmean(aa) |
|-------+----------+-----------|
| apple | 831 | 62.2 |
| grape | -22 | 13.75 |
#+END:
Understand input hline
#+BEGIN: aggregate :table "hline-hash.json:(json)" :cols "hline bb" :hline "1"
| hline | bb |
|-------+-----|
| 0 | 654 |
| 0 | -78 |
|-------+-----|
| 1 | +45 |
| 1 | +35 |
| 1 | +66 |
| 1 | +91 |
|-------+-----|
| 2 | -15 |
| 2 | 7 |
| 2 | 4 |
#+END:
JSON in distant Org file
#+BEGIN: aggregate :table "distant-tests.org:distantjson(json)" :cols "$2 vsum($1)"
| $2 | vsum($1) |
|------+----------|
| univ | 42 |
| jes | 555 |
| ne | -176 |
#+END:
================================================
FILE: tests/wizard-test.el
================================================
;;; wizard-test.el --- Create unit tests for OrgAggregate wizard -*- coding:utf-8; lexical-binding: t;-*-
;; Copyright (C) 2013-2026 Thierry Banel
;;
;; org-aggregate is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; org-aggregate is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see .
;;; Commentary:
;; A Lisp function to record a wizard unit test.
;; - put the cursor somewhere in an Org Mode file,
;; - call M-x orgtbl-aggregate-bench-create
;; - call the wizard with
;; C-c C-x x aggregate
;; - use the wizard as usual, your key-strokes are recorded while you type,
;; - when done, hit the & key,
;; this insert a keyboard macro in the Org Mode file,
;; - the keyboard macro can be executed anytime with C-x e
;; to replicate your actions.
(defun orgtbl-aggregate-bench-create ()
"Interactively create a bench.
When done, type &."
(interactive)
(local-set-key "&" 'orgtbl-aggregate-bench-collect)
(message "use the wizard, then type & when done")
(kmacro-start-macro nil))
(defun orgtbl-aggregate-bench-collect ()
"Called when typing & to close the interactive wizard session.
Do not call it directly."
(interactive)
(kmacro-end-macro 1)
(insert "(execute-kbd-macro (kbd \"\n")
(insert (key-description (kmacro--keys (kmacro last-kbd-macro))))
(insert "\"))\n"))
================================================
FILE: tests/wizardtests.org
================================================
# -*- coding:utf-8; -*-
#+TITLE: Orgtbl Aggregate wizard unit tests
Copyright (C) 2013-2026 Thierry Banel
org-aggregate is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
org-aggregate is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see .
* How to run?
Running all tests should not change anything to this page.
Run this script to complete all the unit tests in a disposable
buffer. When done, the buffer and the original, untouched
~wizardtest.org~, are compared, stopping at the first difference.
#+begin_src elisp :results none
;; define a utility to run a unit test
(defun orgtbl-aggregate-wizard-test (&rest args)
(execute-kbd-macro
(kbd
(mapconcat
#'identity
(cl-loop
for a in args
if (symbolp a)
do (unless (memq a '(:init :isid :orgid :file :name :params :slice
:precompute :cols :cond :hline :post))
(user-error "tag %s not recognized" a))
else
collect a)
"\n"))))
;; display this buffer in a window occupying half the frame
(delete-other-windows)
(goto-char (point-min))
(org-cycle '(64))
(split-window-right)
;; Make a new buffer and fill it with the content of unittests.org
(let ((f (buffer-file-name)))
(switch-to-buffer "disposable-wizardtests.org")
(erase-buffer)
(insert-file f))
(org-mode)
(org-cycle '(64))
;; Clean results from prior tests
(save-excursion
(goto-char (point-min))
(replace-regexp
(rx
bol "#+begin: " (* not-newline) "\n"
(*? (* any) "\n")
"#+end:" (* any) "\n")
""))
;; call all wizard invocations
(goto-char (point-min))
(org-next-visible-heading 2)
;;(while (search-forward "(execute-kbd-macro" nil t)
;; (beginning-of-line)
;; (forward-sexp)
;; (forward-line)
;; (beginning-of-line)
;; (eval-last-sexp nil))
(while (re-search-forward
(rx "orgtbl-aggregate-wizard-test"
(*? (* any) "\n")
"#+end_src" (* any) "\n")
nil t)
(org-ctrl-c-ctrl-c))
;; Compare the disposable buffer with the reference wiz1.org
(goto-char (point-min))
(compare-windows nil)
#+end_src
* test1
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x aggr "
:isid "n"
:file "unitt "
:name "hlin "
:params ""
:slice ""
:precompute "y * 1 0 ; ' y 1 0 ' "
:cols "h l i n e SPC f SPC v s u m ( y ) SPC v s u m ( y 1 0 ) "
:cond ""
:hline ""
:post ""
)
#+end_src
#+BEGIN: aggregate :table "unittests.org:hlinetable" :cols "hline f vsum(y) vsum(y10)" :precompute "y*10;'y10'"
| hline | f | vsum(y) | vsum(y10) |
|-------+---+----------------+-----------------------|
| 0 | 0 | 22 | 220 |
| 1 | 1 | 6 | 60 |
| 2 | 1 | a + 16 | 10 a + 160 |
| 3 | 1 | 10 a + b + b^2 | 100 a + 10 b + 10 b^2 |
#+END:
* a table
#+name: tagqty
| tag | val | qty |
|-----+------+------|
| ccc | 9.91 | 4.4 |
| a | 5.10 | -4.0 |
| ccc | 3.86 | 9.9 |
| bb | 6.13 | 0.4 |
| bb | 8.30 | -6.5 |
| a | 4.58 | 8.8 |
| bb | 0.68 | 4.6 |
| a | 0.96 | 5.6 |
| ccc | 2.43 | 9.0 |
| a | 5.34 | -2.7 |
| a | 4.51 | -6.7 |
| bb | 3.51 | 5.5 |
| a | 9.24 | 3.7 |
| ccc | 0.19 | -6.7 |
| ccc | 2.84 | 3.0 |
| ccc | 3.04 | -5.3 |
| bb | 9.78 | 3.1 |
| ccc | 7.37 | -7.0 |
| bb | 0.91 | 4.8 |
| ccc | 5.28 | -2.4 |
| bb | 1.43 | 9.2 |
| a | 0.11 | 5.1 |
* test 2
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "n"
:file ""
:name "t a g q "
:params ""
:slice "[ : ] "
:precompute ""
:cols "t a g SPC v s u m ( v a l ) SPC v s u m ( q t y ) SPC c o u n t ( ) "
:cond "( < = SPC ( s t r i n g - t o - n u m b e r SPC v a l ) SPC 9 . 0 0 ) ) "
:hline ""
:post ""
)
#+end_src
#+BEGIN: aggregate :table "tagqty[0:14]" :cols "tag vsum(val) vsum(qty) count()" :cond (<= (string-to-number val) 9.0)
| tag | vsum(val) | vsum(qty) | count() |
|-----+-----------+-----------+---------|
| a | 20.49 | 1. | 5 |
| ccc | 6.29 | 18.9 | 2 |
| bb | 18.62 | 4. | 4 |
#+END:
* test :post
#+name: addendrow
#+begin_src elisp :var table=*this* :colnames t
(message "TABLE = %S" table)
(nconc
table
(list
'hline
(cl-loop
for i from 1 to (length (car table))
collect (format "x%d" i))))
#+end_src
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "n"
:file ""
:name "t a g q "
:params ""
:slice ""
:precompute ""
:cols "t a g SPC v m e a n ( v a l ) ; f 2 SPC v m e a n ( q t y ) ; f 2 "
:cond ""
:hline ""
:post "a d d e n d r o w "
)
#+end_src
#+BEGIN: aggregate :table "tagqty" :cols "tag vmean(val);f2 vmean(qty);f2" :post "addendrow"
| tag | vmean(val) | vmean(qty) |
|-----+------------+------------|
| ccc | 4.37 | 0.61 |
| a | 4.26 | 1.40 |
| bb | 4.39 | 3.01 |
|-----+------------+------------|
| x1 | x2 | x3 |
#+END:
* test :hline
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "n"
:file ""
:name "t a g q "
:params ""
:slice ""
:precompute "f l o o r ( v a l ) ; ' v f ' "
:cols "t a g ; ^ a SPC v f ; ^ n SPC v m e a n ( q t y ) SPC c o u n t ( ) "
:cond ""
:hline "1 "
:post ""
)
#+end_src
#+BEGIN: aggregate :table "tagqty" :cols "tag;^a vf;^n vmean(qty) count()" :hline "1" :precompute "floor(val);'vf'"
| tag | vf | vmean(qty) | count() |
|-----+----+------------+---------|
| a | 0 | 5.35 | 2 |
| a | 4 | 1.05 | 2 |
| a | 5 | -3.35 | 2 |
| a | 9 | 3.7 | 1 |
|-----+----+------------+---------|
| bb | 0 | 4.7 | 2 |
| bb | 1 | 9.2 | 1 |
| bb | 3 | 5.5 | 1 |
| bb | 6 | 0.4 | 1 |
| bb | 8 | -6.5 | 1 |
| bb | 9 | 3.1 | 1 |
|-----+----+------------+---------|
| ccc | 0 | -6.7 | 1 |
| ccc | 2 | 6. | 2 |
| ccc | 3 | 2.3 | 2 |
| ccc | 5 | -2.4 | 1 |
| ccc | 7 | -7. | 1 |
| ccc | 9 | 4.4 | 1 |
#+END:
* test CSV
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "n"
:file "g e o g r c "
:name ""
:params ""
:slice ""
:precompute ""
:cols "$ 2 SPC v s u m ( $ 3 ) ; f 0 SPC v m e a n ( $ 4 ) ; f 0 ; ^ n SPC c o u n t ( ) "
:cond ""
:hline ""
:post ""
)
#+end_src
#+BEGIN: aggregate :table "geography-a.csv:(csv)" :cols "$2 vsum($3);f0 vmean($4);f0;^n count()"
| $2 | vsum($3) | vmean($4) | count() |
|----------------+-----------+-----------+---------|
| Israel | 4577871 | 4500492 | 1 |
| Taiwan | 4570576 | 4522439 | 1 |
| Cameroon | 4859198 | 4692347 | 1 |
| Afghanistan | 4862586 | 4712793 | 1 |
| Australia | 10663074 | 5247013. | 2 |
| Kenya | 5772121 | 5560131 | 1 |
| South Africa | 11501203 | 5649307. | 2 |
| Ethiopia | 5961711 | 5681609 | 1 |
| Myanmar | 5829964 | 5706310 | 1 |
| Ivory Coast | 6054358 | 5859424 | 1 |
| Singapore | 6167759 | 6115882 | 1 |
| Spain | 12577695 | 6265703 | 2 |
| Saudi Arabia | 12980333 | 6386230. | 2 |
| Canada | 6513813 | 6450438 | 1 |
| Sudan | 6778168 | 6526345 | 1 |
| Chile | 6973392 | 6953542 | 1 |
| Vietnam | 15392473 | 7481426. | 2 |
| Hong Kong | 7791531 | 7716372 | 1 |
| Iraq | 8154140 | 7911328 | 1 |
| United States | 7966324 | 8100605 | 1 |
| Tanzania | 8529744 | 8188494 | 1 |
| Malaysia | 8980578 | 8832827 | 1 |
| Russia | 18346030 | 9144834 | 2 |
| China | 264183191 | 9260336. | 28 |
| Iran | 9738111 | 9606808 | 1 |
| Angola | 10049628 | 9648709 | 1 |
| United Kingdom | 9818142 | 9723207 | 1 |
| South Korea | 10059272 | 9991484 | 1 |
| Nigeria | 21763797 | 10518043 | 2 |
| Turkey | 21777496 | 10765670 | 2 |
| Mexico | 33697146 | 11061345. | 3 |
| Thailand | 11415533 | 11195528 | 1 |
| France | 11352823 | 11304387 | 1 |
| Peru | 11529982 | 11388304 | 1 |
| Indonesia | 11628728 | 11478423 | 1 |
| Colombia | 11779275 | 11660428 | 1 |
| Brazil | 48284586 | 11950654. | 4 |
| India | 140092165 | 13696246 | 10 |
| Egypt | 28897566 | 14165193. | 2 |
| Bangladesh | 30197563 | 14745963. | 2 |
| Philippines | 15211511 | 14917612 | 1 |
| Argentina | 15714124 | 15634092 | 1 |
| Pakistan | 32895483 | 15998680. | 2 |
| DR Congo | 17815364 | 17025505 | 1 |
| Japan | 71117892 | 17814626. | 4 |
#+END:
* test JSON
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "n"
:file "g e o g r j "
:name ""
:params ""
:slice ""
:precompute ""
:cols "$ 2 SPC v m i n ( $ 3 ) SPC v m a x ( $ 3 ) ; ^ n "
:cond "( > = SPC $ 4 SPC 6 0 0 0 0 0 0 ) "
:hline ""
:post ""
)
#+end_src
#+BEGIN: aggregate :table "geography-a.json:(json)" :cols "$2 vmin($3) vmax($3);^n" :cond (>= $4 6000000)
| $2 | vmin($3) | vmax($3) |
|----------------+----------+----------|
| Singapore | 6167759 | 6167759 |
| South Africa | 6436807 | 6436807 |
| Canada | 6513813 | 6513813 |
| Sudan | 6778168 | 6778168 |
| Spain | 6826620 | 6826620 |
| Chile | 6973392 | 6973392 |
| Hong Kong | 7791531 | 7791531 |
| Saudi Arabia | 7964688 | 7964688 |
| United States | 7966324 | 7966324 |
| Iraq | 8154140 | 8154140 |
| Tanzania | 8529744 | 8529744 |
| Malaysia | 8980578 | 8980578 |
| Iran | 9738111 | 9738111 |
| Vietnam | 9798896 | 9798896 |
| United Kingdom | 9818142 | 9818142 |
| Angola | 10049628 | 10049628 |
| South Korea | 10059272 | 10059272 |
| France | 11352823 | 11352823 |
| Thailand | 11415533 | 11415533 |
| Peru | 11529982 | 11529982 |
| Indonesia | 11628728 | 11628728 |
| Colombia | 11779275 | 11779275 |
| Russia | 12768223 | 12768223 |
| Philippines | 15211511 | 15211511 |
| Argentina | 15714124 | 15714124 |
| Turkey | 16211581 | 16211581 |
| Nigeria | 17124998 | 17124998 |
| DR Congo | 17815364 | 17815364 |
| Pakistan | 14835678 | 18059805 |
| Mexico | 22831373 | 22831373 |
| Brazil | 6360069 | 23045227 |
| Egypt | 23095986 | 23095986 |
| Bangladesh | 24561693 | 24561693 |
| China | 6242353 | 30365228 |
| India | 7498726 | 34598951 |
| Japan | 9544065 | 37131070 |
#+END:
* test ID
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x a g g "
:isid "y"
:orgid "55ab27a2 "
:params ""
:slice ""
:precompute ""
:cols "ref SPC vsum(val) SPC count() "
:cond ""
:hline ""
:post ""
)
#+end_src
#+BEGIN: aggregate :table "55ab27a2-c44b-4a14-9ba4-f6879375207d" :cols "ref vsum(val) count()"
| ref | vsum(val) | count() |
|------+-----------+---------|
| S | 2490.9 | 4 |
| TT | 2352.3 | 6 |
| UUU | 1643.8 | 4 |
| VVVV | 3144.5 | 5 |
#+END:
* transpose
#+name: sunnydays
| feat | | mon | tue | wed | thu | fri | sat | sun |
|------+---+-----+-----+-----+-----+-----+-----+-----|
| rain | | no | yes | yes | no | no | no | no |
| temp | | 23 | 15 | 13 | 16 | 19 | 22 | 25 |
| wind | | 10 | 0 | 10 | 20 | 20 | 25 | 15 |
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x t r a n "
:isid "n"
:file ""
:name "C-a SPC C-a C-k s u n n y d "
:params ""
:slice ""
:cols ""
:cond ""
:post ""
)
#+end_src
#+BEGIN: transpose :table "sunnydays" :cols ""
| feat | | rain | temp | wind |
|------+---+------+------+------|
| mon | | no | 23 | 10 |
| tue | | yes | 15 | 0 |
| wed | | yes | 13 | 10 |
| thu | | no | 16 | 20 |
| fri | | no | 19 | 20 |
| sat | | no | 22 | 25 |
| sun | | no | 25 | 15 |
#+END:
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-u C-c C-x x tran "
:isid "n"
:file ""
:name "C-a SPC C-a C-k s u n n y d "
:params ""
:slice ""
:cols "feat SPC sun SPC mon SPC sat "
:cond ""
:post ""
)
#+end_src
#+BEGIN: transpose :table "sunnydays" :cols "feat sun mon sat"
| feat | | rain | temp | wind |
| sun | | no | 25 | 15 |
| mon | | no | 23 | 10 |
| sat | | no | 22 | 25 |
#+END:
* update preserving #+tblfm: formula
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-c C-x x aggreg "
:file "C-a SPC C-a C-k distant-tests.org "
:name "C-a SPC C-a C-k distanttabl "
:slice ""
:cols ""
)
#+end_src
#+BEGIN: aggregate :table "distant-tests.org:distanttable[0:10]" :cols "tag vsum(val)" :formula "$3=$2*10"
| tag | vsum(val) | | |
|-----+-----------+--------+---------|
| A | 210.76 | 2107.6 | 12107.6 |
| BB | 257.22 | 2572.2 | 12572.2 |
| CCC | 92.06 | 920.6 | 10920.6 |
#+TBLFM: $3=$2*10::$4=$3+10000
#+END:
▲
╰──indent in order to update the #+begin: line instead of creating it anew
#+begin_src elisp :results none
(orgtbl-aggregate-wizard-test
:init "C-c C-x x transpose "
:file "C-a SPC C-a C-k distant-tests.org "
:name "C-a SPC C-a C-k distanttabl "
:slice ""
:cols ""
)
#+end_src
#+BEGIN: transpose :table "distant-tests.org:distanttable[0:5]" :cols ""
| tag | | A | BB | CCC | BB | BB + 1000 |
| val | | 11.58 | 98.43 | 87.74 | 87.97 | 1087.97 |
#+TBLFM: $7=$-1+1000
#+END:
▲
╰──indent in order to update the #+begin: line instead of creating it anew