Showing preview only (382K chars total). Download the full file or copy to clipboard to get everything.
Repository: camlistore/go4
Branch: master
Commit: a5071408f32f
Files: 103
Total size: 355.6 KB
Directory structure:
gitextract_mezihb_5/
├── .gitignore
├── .travis.yml
├── AUTHORS
├── LICENSE
├── README.md
├── bytereplacer/
│ ├── bytereplacer.go
│ └── bytereplacer_test.go
├── cloud/
│ ├── cloudlaunch/
│ │ └── cloudlaunch.go
│ └── google/
│ ├── gceutil/
│ │ └── gceutil.go
│ └── gcsutil/
│ └── storage.go
├── ctxutil/
│ └── ctxutil.go
├── errorutil/
│ └── highlight.go
├── fault/
│ └── fault.go
├── go.mod
├── go.sum
├── go4test/
│ └── cloudlaunch/
│ └── serve_on_cloud.go
├── jsonconfig/
│ ├── eval.go
│ ├── jsonconfig.go
│ ├── jsonconfig_test.go
│ └── testdata/
│ ├── boolenv.json
│ ├── include1.json
│ ├── include1bis.json
│ ├── include2.json
│ ├── listexpand.json
│ ├── loop1.json
│ └── loop2.json
├── legal/
│ ├── legal.go
│ └── legal_test.go
├── lock/
│ ├── .gitignore
│ ├── lock.go
│ ├── lock_plan9.go
│ ├── lock_sigzero.go
│ ├── lock_test.go
│ ├── lock_unix.go
│ └── lock_windows.go
├── media/
│ └── heif/
│ ├── bmff/
│ │ └── bmff.go
│ ├── dumpheif/
│ │ └── dumpheif.go
│ ├── heif.go
│ ├── heif_test.go
│ └── testdata/
│ ├── park.heic
│ └── rotate.heic
├── must/
│ └── must.go
├── net/
│ └── throttle/
│ └── throttle.go
├── oauthutil/
│ └── oauth.go
├── osutil/
│ └── osutil.go
├── readerutil/
│ ├── bufreaderat.go
│ ├── bufreaderat_test.go
│ ├── countingreader.go
│ ├── fakeseeker.go
│ ├── fakeseeker_test.go
│ ├── multireaderat.go
│ ├── multireaderat_test.go
│ ├── readersize.go
│ ├── readersize_test.go
│ ├── readerutil.go
│ ├── readerutil_test.go
│ └── singlereader/
│ ├── opener.go
│ └── opener_test.go
├── reflectutil/
│ └── swapper.go
├── rollsum/
│ ├── rollsum.go
│ └── rollsum_test.go
├── sort/
│ ├── example_interface_test.go
│ ├── example_keys_test.go
│ ├── example_multi_test.go
│ ├── example_slice_test.go
│ ├── example_test.go
│ ├── example_wrapper_test.go
│ ├── export_test.go
│ ├── genzfunc.go
│ ├── search.go
│ ├── search_test.go
│ ├── sort.go
│ ├── sort_test.go
│ └── zfuncversion.go
├── strutil/
│ ├── intern.go
│ ├── strconv.go
│ ├── strutil.go
│ └── strutil_test.go
├── syncutil/
│ ├── gate.go
│ ├── group.go
│ ├── once.go
│ ├── once_test.go
│ ├── sem.go
│ ├── sem_test.go
│ ├── singleflight/
│ │ ├── singleflight.go
│ │ └── singleflight_test.go
│ ├── syncdebug/
│ │ ├── syncdebug.go
│ │ └── syncdebug_test.go
│ └── syncutil.go
├── testing/
│ └── functest/
│ ├── functest.go
│ └── functest_test.go
├── types/
│ ├── types.go
│ └── types_test.go
├── wkfs/
│ ├── gcs/
│ │ ├── gcs.go
│ │ └── gcs_test.go
│ └── wkfs.go
├── writerutil/
│ ├── writerutil.go
│ └── writerutil_test.go
├── xdgdir/
│ ├── example_test.go
│ ├── xdgdir.go
│ └── xdgdir_test.go
└── ziputil/
├── ziputil.go
└── ziputil_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof
================================================
FILE: .travis.yml
================================================
language: go
arch:
- amd64
- ppc64le
go:
- "1.13.x"
- tip
go_import_path: go4.org
before_install:
- go mod tidy
- git diff --exit-code go.mod
- git diff --exit-code go.sum
- go mod download
================================================
FILE: AUTHORS
================================================
# This is the official list of go4 authors for copyright purposes.
# This is distinct from the CONTRIBUTORS file, which is the list of
# people who have contributed, even if they don't own the copyright on
# their work.
Mathieu Lonjaret <mathieu.lonjaret@gmail.com>
Daniel Theophanes <kardianos@gmail.com>
Google
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
# go4
[](https://travis-ci.org/go4org/go4 "Travis CI")
[go4.org](http://go4.org) is a collection of packages for
Go programmers.
They started out living in [Perkeep](https://perkeep.org)'s repo
and elsewhere but they have nothing to do with Perkeep, so we're
moving them here.
## Details
* **single repo**. go4 is a single repo. That means things can be
changed and rearranged globally atomically with ease and
confidence.
* **no backwards compatibility**. go4 makes no backwards compatibility
promises. If you want to use go4, vendor it. And next time you
update your vendor tree, update to the latest API if things in go4
changed. The plan is to eventually provide tools to make this
easier.
* **forward progress** because we have no backwards compatibility,
it's always okay to change things to make things better. That also
means the bar for contributions is lower. We don't have to get the
API 100% correct in the first commit.
* **no Go version policy** go4 packages are usually built and tested
with the latest Go stable version. However, go4 has no overarching
version policy; each package can declare its own set of supported
Go versions.
* **code review** contributions must be code-reviewed. We're trying
out Gerrithub, to see if we can find a mix of Github Pull Requests
and Gerrit that works well for many people. We'll see.
* **CLA compliant** contributors must agree to the Google CLA (the
same as Go itself). This ensures we can move things into Go as
necessary in the future. It also makes lawyers at various
companies happy. The CLA is **not** a copyright *assignment*; you
retain the copyright on your work. The CLA just says that your
work is open source and you have permission to open source it. See
https://golang.org/doc/contribute.html#cla
* **docs, tests, portability** all code should be documented in the
normal Go style, have tests, and be portable to different
operating systems and architectures. We'll try to get builders in
place to help run the tests on different OS/arches. For now we
have Travis at least.
## Contact
For any question, or communication when a Github issue is not appropriate,
please contact the [Perkeep mailing
list](https://groups.google.com/forum/#!forum/perkeep).
================================================
FILE: bytereplacer/bytereplacer.go
================================================
/*
Copyright 2015 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package bytereplacer provides a utility for replacing parts of byte slices.
package bytereplacer // import "go4.org/bytereplacer"
import "bytes"
// Replacer replaces a list of strings with replacements.
// It is safe for concurrent use by multiple goroutines.
type Replacer struct {
r replacer
}
// replacer is the interface that a replacement algorithm needs to implement.
type replacer interface {
// Replace performs all replacements, in-place if possible.
Replace(s []byte) []byte
}
// New returns a new Replacer from a list of old, new string pairs.
// Replacements are performed in order, without overlapping matches.
func New(oldnew ...string) *Replacer {
if len(oldnew)%2 == 1 {
panic("bytes.NewReplacer: odd argument count")
}
allNewBytes := true
for i := 0; i < len(oldnew); i += 2 {
if len(oldnew[i]) != 1 {
return &Replacer{r: makeGenericReplacer(oldnew)}
}
if len(oldnew[i+1]) != 1 {
allNewBytes = false
}
}
if allNewBytes {
r := byteReplacer{}
for i := range r {
r[i] = byte(i)
}
// The first occurrence of old->new map takes precedence
// over the others with the same old string.
for i := len(oldnew) - 2; i >= 0; i -= 2 {
o := oldnew[i][0]
n := oldnew[i+1][0]
r[o] = n
}
return &Replacer{r: &r}
}
return &Replacer{r: makeGenericReplacer(oldnew)}
}
// Replace performs all replacements in-place on s. If the capacity
// of s is not sufficient, a new slice is allocated, otherwise Replace
// returns s.
func (r *Replacer) Replace(s []byte) []byte {
return r.r.Replace(s)
}
type trieNode struct {
value []byte
priority int
prefix []byte
next *trieNode
table []*trieNode
}
func (t *trieNode) add(key, val []byte, priority int, r *genericReplacer) {
if len(key) == 0 {
if t.priority == 0 {
t.value = val
t.priority = priority
}
return
}
if len(t.prefix) > 0 {
// Need to split the prefix among multiple nodes.
var n int // length of the longest common prefix
for ; n < len(t.prefix) && n < len(key); n++ {
if t.prefix[n] != key[n] {
break
}
}
if n == len(t.prefix) {
t.next.add(key[n:], val, priority, r)
} else if n == 0 {
// First byte differs, start a new lookup table here. Looking up
// what is currently t.prefix[0] will lead to prefixNode, and
// looking up key[0] will lead to keyNode.
var prefixNode *trieNode
if len(t.prefix) == 1 {
prefixNode = t.next
} else {
prefixNode = &trieNode{
prefix: t.prefix[1:],
next: t.next,
}
}
keyNode := new(trieNode)
t.table = make([]*trieNode, r.tableSize)
t.table[r.mapping[t.prefix[0]]] = prefixNode
t.table[r.mapping[key[0]]] = keyNode
t.prefix = nil
t.next = nil
keyNode.add(key[1:], val, priority, r)
} else {
// Insert new node after the common section of the prefix.
next := &trieNode{
prefix: t.prefix[n:],
next: t.next,
}
t.prefix = t.prefix[:n]
t.next = next
next.add(key[n:], val, priority, r)
}
} else if t.table != nil {
// Insert into existing table.
m := r.mapping[key[0]]
if t.table[m] == nil {
t.table[m] = new(trieNode)
}
t.table[m].add(key[1:], val, priority, r)
} else {
t.prefix = key
t.next = new(trieNode)
t.next.add(nil, val, priority, r)
}
}
func (r *genericReplacer) lookup(s []byte, ignoreRoot bool) (val []byte, keylen int, found bool) {
// Iterate down the trie to the end, and grab the value and keylen with
// the highest priority.
bestPriority := 0
node := &r.root
n := 0
for node != nil {
if node.priority > bestPriority && !(ignoreRoot && node == &r.root) {
bestPriority = node.priority
val = node.value
keylen = n
found = true
}
if len(s) == 0 {
break
}
if node.table != nil {
index := r.mapping[s[0]]
if int(index) == r.tableSize {
break
}
node = node.table[index]
s = s[1:]
n++
} else if len(node.prefix) > 0 && bytes.HasPrefix(s, node.prefix) {
n += len(node.prefix)
s = s[len(node.prefix):]
node = node.next
} else {
break
}
}
return
}
// genericReplacer is the fully generic algorithm.
// It's used as a fallback when nothing faster can be used.
type genericReplacer struct {
root trieNode
// tableSize is the size of a trie node's lookup table. It is the number
// of unique key bytes.
tableSize int
// mapping maps from key bytes to a dense index for trieNode.table.
mapping [256]byte
}
func makeGenericReplacer(oldnew []string) *genericReplacer {
r := new(genericReplacer)
// Find each byte used, then assign them each an index.
for i := 0; i < len(oldnew); i += 2 {
key := oldnew[i]
for j := 0; j < len(key); j++ {
r.mapping[key[j]] = 1
}
}
for _, b := range r.mapping {
r.tableSize += int(b)
}
var index byte
for i, b := range r.mapping {
if b == 0 {
r.mapping[i] = byte(r.tableSize)
} else {
r.mapping[i] = index
index++
}
}
// Ensure root node uses a lookup table (for performance).
r.root.table = make([]*trieNode, r.tableSize)
for i := 0; i < len(oldnew); i += 2 {
r.root.add([]byte(oldnew[i]), []byte(oldnew[i+1]), len(oldnew)-i, r)
}
return r
}
func (r *genericReplacer) Replace(s []byte) []byte {
var last int
var prevMatchEmpty bool
dst := s[:0]
grown := false
for i := 0; i <= len(s); {
// Fast path: s[i] is not a prefix of any pattern.
if i != len(s) && r.root.priority == 0 {
index := int(r.mapping[s[i]])
if index == r.tableSize || r.root.table[index] == nil {
i++
continue
}
}
// Ignore the empty match iff the previous loop found the empty match.
val, keylen, match := r.lookup(s[i:], prevMatchEmpty)
prevMatchEmpty = match && keylen == 0
if match {
dst = append(dst, s[last:i]...)
if diff := len(val) - keylen; grown || diff < 0 {
dst = append(dst, val...)
i += keylen
} else if diff <= cap(s)-len(s) {
// The replacement is larger than the original, but can still fit in the original buffer.
copy(s[i+len(val):cap(dst)], s[i+keylen:])
dst = append(dst, val...)
s = s[:len(s)+diff]
i += len(val)
} else {
// The output will grow larger than the original buffer. Allocate a new one.
grown = true
newDst := make([]byte, len(dst), cap(dst)+diff)
copy(newDst, dst)
dst = newDst
dst = append(dst, val...)
i += keylen
}
last = i
continue
}
i++
}
if last != len(s) {
dst = append(dst, s[last:]...)
}
return dst
}
// byteReplacer is the implementation that's used when all the "old"
// and "new" values are single ASCII bytes.
// The array contains replacement bytes indexed by old byte.
type byteReplacer [256]byte
func (r *byteReplacer) Replace(s []byte) []byte {
for i, b := range s {
s[i] = r[b]
}
return s
}
================================================
FILE: bytereplacer/bytereplacer_test.go
================================================
/*
Copyright 2015 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package bytereplacer
import (
"bytes"
"strings"
"testing"
)
var htmlEscaper = New(
"&", "&",
"<", "<",
">", ">",
`"`, """,
"'", "'",
)
var htmlUnescaper = New(
"&", "&",
"<", "<",
">", ">",
""", `"`,
"'", "'",
)
var capitalLetters = New("a", "A", "b", "B")
func TestReplacer(t *testing.T) {
type testCase struct {
r *Replacer
in, out string
}
var testCases []testCase
// str converts 0xff to "\xff". This isn't just string(b) since that converts to UTF-8.
str := func(b byte) string {
return string([]byte{b})
}
var s []string
// inc maps "\x00"->"\x01", ..., "a"->"b", "b"->"c", ..., "\xff"->"\x00".
s = nil
for i := 0; i < 256; i++ {
s = append(s, str(byte(i)), str(byte(i+1)))
}
inc := New(s...)
// Test cases with 1-byte old strings, 1-byte new strings.
testCases = append(testCases,
testCase{capitalLetters, "brad", "BrAd"},
testCase{capitalLetters, strings.Repeat("a", (32<<10)+123), strings.Repeat("A", (32<<10)+123)},
testCase{capitalLetters, "", ""},
testCase{inc, "brad", "csbe"},
testCase{inc, "\x00\xff", "\x01\x00"},
testCase{inc, "", ""},
testCase{New("a", "1", "a", "2"), "brad", "br1d"},
)
// repeat maps "a"->"a", "b"->"bb", "c"->"ccc", ...
s = nil
for i := 0; i < 256; i++ {
n := i + 1 - 'a'
if n < 1 {
n = 1
}
s = append(s, str(byte(i)), strings.Repeat(str(byte(i)), n))
}
repeat := New(s...)
// Test cases with 1-byte old strings, variable length new strings.
testCases = append(testCases,
testCase{htmlEscaper, "No changes", "No changes"},
testCase{htmlEscaper, "I <3 escaping & stuff", "I <3 escaping & stuff"},
testCase{htmlEscaper, "&&&", "&&&"},
testCase{htmlEscaper, "", ""},
testCase{repeat, "brad", "bbrrrrrrrrrrrrrrrrrradddd"},
testCase{repeat, "abba", "abbbba"},
testCase{repeat, "", ""},
testCase{New("a", "11", "a", "22"), "brad", "br11d"},
)
// The remaining test cases have variable length old strings.
testCases = append(testCases,
testCase{htmlUnescaper, "&amp;", "&"},
testCase{htmlUnescaper, "<b>HTML's neat</b>", "<b>HTML's neat</b>"},
testCase{htmlUnescaper, "", ""},
testCase{New("a", "1", "a", "2", "xxx", "xxx"), "brad", "br1d"},
testCase{New("a", "1", "aa", "2", "aaa", "3"), "aaaa", "1111"},
testCase{New("aaa", "3", "aa", "2", "a", "1"), "aaaa", "31"},
)
// gen1 has multiple old strings of variable length. There is no
// overall non-empty common prefix, but some pairwise common prefixes.
gen1 := New(
"aaa", "3[aaa]",
"aa", "2[aa]",
"a", "1[a]",
"i", "i",
"longerst", "most long",
"longer", "medium",
"long", "short",
"xx", "xx",
"x", "X",
"X", "Y",
"Y", "Z",
)
testCases = append(testCases,
testCase{gen1, "fooaaabar", "foo3[aaa]b1[a]r"},
testCase{gen1, "long, longerst, longer", "short, most long, medium"},
testCase{gen1, "xxxxx", "xxxxX"},
testCase{gen1, "XiX", "YiY"},
testCase{gen1, "", ""},
)
// gen2 has multiple old strings with no pairwise common prefix.
gen2 := New(
"roses", "red",
"violets", "blue",
"sugar", "sweet",
)
testCases = append(testCases,
testCase{gen2, "roses are red, violets are blue...", "red are red, blue are blue..."},
testCase{gen2, "", ""},
)
// gen3 has multiple old strings with an overall common prefix.
gen3 := New(
"abracadabra", "poof",
"abracadabrakazam", "splat",
"abraham", "lincoln",
"abrasion", "scrape",
"abraham", "isaac",
)
testCases = append(testCases,
testCase{gen3, "abracadabrakazam abraham", "poofkazam lincoln"},
testCase{gen3, "abrasion abracad", "scrape abracad"},
testCase{gen3, "abba abram abrasive", "abba abram abrasive"},
testCase{gen3, "", ""},
)
// foo{1,2,3,4} have multiple old strings with an overall common prefix
// and 1- or 2- byte extensions from the common prefix.
foo1 := New(
"foo1", "A",
"foo2", "B",
"foo3", "C",
)
foo2 := New(
"foo1", "A",
"foo2", "B",
"foo31", "C",
"foo32", "D",
)
foo3 := New(
"foo11", "A",
"foo12", "B",
"foo31", "C",
"foo32", "D",
)
foo4 := New(
"foo12", "B",
"foo32", "D",
)
testCases = append(testCases,
testCase{foo1, "fofoofoo12foo32oo", "fofooA2C2oo"},
testCase{foo1, "", ""},
testCase{foo2, "fofoofoo12foo32oo", "fofooA2Doo"},
testCase{foo2, "", ""},
testCase{foo3, "fofoofoo12foo32oo", "fofooBDoo"},
testCase{foo3, "", ""},
testCase{foo4, "fofoofoo12foo32oo", "fofooBDoo"},
testCase{foo4, "", ""},
)
// genAll maps "\x00\x01\x02...\xfe\xff" to "[all]", amongst other things.
allBytes := make([]byte, 256)
for i := range allBytes {
allBytes[i] = byte(i)
}
allString := string(allBytes)
genAll := New(
allString, "[all]",
"\xff", "[ff]",
"\x00", "[00]",
)
testCases = append(testCases,
testCase{genAll, allString, "[all]"},
testCase{genAll, "a\xff" + allString + "\x00", "a[ff][all][00]"},
testCase{genAll, "", ""},
)
// Test cases with empty old strings.
blankToX1 := New("", "X")
blankToX2 := New("", "X", "", "")
blankHighPriority := New("", "X", "o", "O")
blankLowPriority := New("o", "O", "", "X")
blankNoOp1 := New("", "")
blankNoOp2 := New("", "", "", "A")
blankFoo := New("", "X", "foobar", "R", "foobaz", "Z")
testCases = append(testCases,
testCase{blankToX1, "foo", "XfXoXoX"},
testCase{blankToX1, "", "X"},
testCase{blankToX2, "foo", "XfXoXoX"},
testCase{blankToX2, "", "X"},
testCase{blankHighPriority, "oo", "XOXOX"},
testCase{blankHighPriority, "ii", "XiXiX"},
testCase{blankHighPriority, "oiio", "XOXiXiXOX"},
testCase{blankHighPriority, "iooi", "XiXOXOXiX"},
testCase{blankHighPriority, "", "X"},
testCase{blankLowPriority, "oo", "OOX"},
testCase{blankLowPriority, "ii", "XiXiX"},
testCase{blankLowPriority, "oiio", "OXiXiOX"},
testCase{blankLowPriority, "iooi", "XiOOXiX"},
testCase{blankLowPriority, "", "X"},
testCase{blankNoOp1, "foo", "foo"},
testCase{blankNoOp1, "", ""},
testCase{blankNoOp2, "foo", "foo"},
testCase{blankNoOp2, "", ""},
testCase{blankFoo, "foobarfoobaz", "XRXZX"},
testCase{blankFoo, "foobar-foobaz", "XRX-XZX"},
testCase{blankFoo, "", "X"},
)
// single string replacer
abcMatcher := New("abc", "[match]")
testCases = append(testCases,
testCase{abcMatcher, "", ""},
testCase{abcMatcher, "ab", "ab"},
testCase{abcMatcher, "abc", "[match]"},
testCase{abcMatcher, "abcd", "[match]d"},
testCase{abcMatcher, "cabcabcdabca", "c[match][match]d[match]a"},
)
// Issue 6659 cases (more single string replacer)
noHello := New("Hello", "")
testCases = append(testCases,
testCase{noHello, "Hello", ""},
testCase{noHello, "Hellox", "x"},
testCase{noHello, "xHello", "x"},
testCase{noHello, "xHellox", "xx"},
)
// No-arg test cases.
nop := New()
testCases = append(testCases,
testCase{nop, "abc", "abc"},
testCase{nop, "", ""},
)
// Run the test cases.
for i, tc := range testCases {
{
// Replace with len(in) == cap(in)
in := make([]byte, len(tc.in))
copy(in, tc.in)
if s := string(tc.r.Replace(in)); s != tc.out {
t.Errorf("%d. Replace(%q /* len == cap */) = %q, want %q", i, tc.in, s, tc.out)
}
}
{
// Replace with len(in) < cap(in)
in := make([]byte, len(tc.in), len(tc.in)*2)
copy(in, tc.in)
if s := string(tc.r.Replace(in)); s != tc.out {
t.Errorf("%d. Replace(%q /* len < cap */) = %q, want %q", i, tc.in, s, tc.out)
}
}
}
}
func BenchmarkGenericNoMatch(b *testing.B) {
str := []byte(strings.Repeat("A", 100) + strings.Repeat("B", 100))
generic := New("a", "A", "b", "B", "12", "123") // varying lengths forces generic
for i := 0; i < b.N; i++ {
generic.Replace(str)
}
}
func BenchmarkGenericMatch1(b *testing.B) {
str := []byte(strings.Repeat("a", 100) + strings.Repeat("b", 100))
generic := New("a", "A", "b", "B", "12", "123")
for i := 0; i < b.N; i++ {
generic.Replace(str)
}
}
func BenchmarkGenericMatch2(b *testing.B) {
str := bytes.Repeat([]byte("It's <b>HTML</b>!"), 100)
for i := 0; i < b.N; i++ {
htmlUnescaper.Replace(str)
}
}
func benchmarkSingleString(b *testing.B, pattern, text string) {
r := New(pattern, "[match]")
buf := make([]byte, len(text), len(text)*7)
b.SetBytes(int64(len(text)))
b.ResetTimer()
for i := 0; i < b.N; i++ {
copy(buf, text)
r.Replace(buf)
}
}
func BenchmarkSingleMaxSkipping(b *testing.B) {
benchmarkSingleString(b, strings.Repeat("b", 25), strings.Repeat("a", 10000))
}
func BenchmarkSingleLongSuffixFail(b *testing.B) {
benchmarkSingleString(b, "b"+strings.Repeat("a", 500), strings.Repeat("a", 1002))
}
func BenchmarkSingleMatch(b *testing.B) {
benchmarkSingleString(b, "abcdef", strings.Repeat("abcdefghijklmno", 1000))
}
func benchmarkReplacer(b *testing.B, r *Replacer, str string) {
buf := make([]byte, len(str))
b.ResetTimer()
for i := 0; i < b.N; i++ {
copy(buf, str)
r.Replace(buf)
}
}
func BenchmarkByteByteNoMatch(b *testing.B) {
benchmarkReplacer(b, capitalLetters, strings.Repeat("A", 100)+strings.Repeat("B", 100))
}
func BenchmarkByteByteMatch(b *testing.B) {
benchmarkReplacer(b, capitalLetters, strings.Repeat("a", 100)+strings.Repeat("b", 100))
}
func BenchmarkByteStringMatch(b *testing.B) {
benchmarkReplacer(b, htmlEscaper, "<"+strings.Repeat("a", 99)+strings.Repeat("b", 99)+">")
}
func BenchmarkHTMLEscapeNew(b *testing.B) {
benchmarkReplacer(b, htmlEscaper, "I <3 to escape HTML & other text too.")
}
func BenchmarkHTMLEscapeOld(b *testing.B) {
str := "I <3 to escape HTML & other text too."
buf := make([]byte, len(str))
for i := 0; i < b.N; i++ {
copy(buf, str)
oldHTMLEscape(buf)
}
}
// The http package's old HTML escaping function in bytes form.
func oldHTMLEscape(s []byte) []byte {
s = bytes.Replace(s, []byte("&"), []byte("&"), -1)
s = bytes.Replace(s, []byte("<"), []byte("<"), -1)
s = bytes.Replace(s, []byte(">"), []byte(">"), -1)
s = bytes.Replace(s, []byte(`"`), []byte("""), -1)
s = bytes.Replace(s, []byte("'"), []byte("'"), -1)
return s
}
// BenchmarkByteByteReplaces compares byteByteImpl against multiple Replaces.
func BenchmarkByteByteReplaces(b *testing.B) {
str := strings.Repeat("a", 100) + strings.Repeat("b", 100)
for i := 0; i < b.N; i++ {
bytes.Replace(bytes.Replace([]byte(str), []byte{'a'}, []byte{'A'}, -1), []byte{'b'}, []byte{'B'}, -1)
}
}
// BenchmarkByteByteMap compares byteByteImpl against Map.
func BenchmarkByteByteMap(b *testing.B) {
str := strings.Repeat("a", 100) + strings.Repeat("b", 100)
fn := func(r rune) rune {
switch r {
case 'a':
return 'A'
case 'b':
return 'B'
}
return r
}
for i := 0; i < b.N; i++ {
bytes.Map(fn, []byte(str))
}
}
================================================
FILE: cloud/cloudlaunch/cloudlaunch.go
================================================
/*
Copyright 2015 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package cloudlaunch helps binaries run themselves on The Cloud, copying
// themselves to GCE.
package cloudlaunch // import "go4.org/cloud/cloudlaunch"
import (
"encoding/json"
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"path/filepath"
"runtime"
"strings"
"time"
"go4.org/cloud/google/gceutil"
"cloud.google.com/go/compute/metadata"
"cloud.google.com/go/storage"
"golang.org/x/net/context"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
compute "google.golang.org/api/compute/v1"
"google.golang.org/api/googleapi"
"google.golang.org/api/option"
storageapi "google.golang.org/api/storage/v1"
)
func readFile(v string) string {
slurp, err := ioutil.ReadFile(v)
if err != nil {
log.Fatalf("Error reading %s: %v", v, err)
}
return strings.TrimSpace(string(slurp))
}
const baseConfig = `#cloud-config
coreos:
update:
group: stable
reboot-strategy: $REBOOT
units:
- name: $NAME.service
command: start
content: |
[Unit]
Description=$NAME service
After=network.target
[Service]
Type=simple
ExecStartPre=/bin/sh -c 'mkdir -p /opt/bin && /usr/bin/curl --silent -f -o /opt/bin/$NAME $URL?$(date +%s) && chmod +x /opt/bin/$NAME'
ExecStart=/opt/bin/$NAME
RestartSec=10
Restart=always
StartLimitInterval=0
[Install]
WantedBy=network-online.target
`
// RestartPolicy controls whether the binary automatically restarts.
type RestartPolicy int
const (
RestartOnUpdates RestartPolicy = iota
RestartNever
// TODO: more graceful restarts; make systemd own listening on network sockets,
// don't break connections.
)
type Config struct {
// Name is the name of a service to run.
// This is the name of the systemd service (without .service)
// and the name of the GCE instance.
Name string
// RestartPolicy controls whether the binary automatically restarts
// on updates. The zero value means automatic.
RestartPolicy RestartPolicy
// UpdateStrategy sets the CoreOS automatic update strategy, and the
// associated reboots. Possible values are "best-effort", "etcd-lock",
// "reboot", "off", with "best-effort" being the default. See
// https://coreos.com/os/docs/latest/update-strategies.html
UpdateStrategy string
// BinaryBucket and BinaryObject are the GCS bucket and object
// within that bucket containing the Linux binary to download
// on boot and occasionally run. This binary must be public
// (at least for now).
BinaryBucket string
BinaryObject string // defaults to Name
GCEProjectID string
Zone string // defaults to us-central1-f
SSD bool
Scopes []string // any additional scopes
MachineType string
InstanceName string
}
// cloudLaunch is a launch of a Config.
type cloudLaunch struct {
*Config
oauthClient *http.Client
computeService *compute.Service
}
func (c *Config) binaryURL() string {
return "https://storage.googleapis.com/" + c.BinaryBucket + "/" + c.binaryObject()
}
func (c *Config) instName() string { return c.Name } // for now
func (c *Config) zone() string { return strDefault(c.Zone, "us-central1-f") }
func (c *Config) machineType() string { return strDefault(c.MachineType, "g1-small") }
func (c *Config) binaryObject() string { return strDefault(c.BinaryObject, c.Name) }
func (c *Config) updateStrategy() string { return strDefault(c.UpdateStrategy, "best-effort") }
func (c *Config) projectAPIURL() string {
return "https://www.googleapis.com/compute/v1/projects/" + c.GCEProjectID
}
func (c *Config) machineTypeURL() string {
return c.projectAPIURL() + "/zones/" + c.zone() + "/machineTypes/" + c.machineType()
}
func strDefault(a, b string) string {
if a != "" {
return a
}
return b
}
var (
doLaunch = flag.Bool("cloudlaunch", false, "Deploy or update this binary to the cloud. Must be on Linux, for now.")
)
func (c *Config) MaybeDeploy() {
flag.Parse()
if !*doLaunch {
go c.restartLoop()
return
}
defer os.Exit(1) // backup, in case we return without Fatal or os.Exit later
if runtime.GOOS != "linux" || runtime.GOARCH != "amd64" {
log.Fatal("Can only use --cloudlaunch on linux/amd64, for now.")
}
if c.GCEProjectID == "" {
log.Fatal("cloudconfig.GCEProjectID is empty")
}
filename := filepath.Join(os.Getenv("HOME"), "keys", c.GCEProjectID+".key.json")
log.Printf("Using OAuth config from JSON service file: %s", filename)
jwtConf, err := google.JWTConfigFromJSON([]byte(readFile(filename)), append([]string{
storageapi.DevstorageFullControlScope,
compute.ComputeScope,
"https://www.googleapis.com/auth/cloud-platform",
}, c.Scopes...)...)
if err != nil {
log.Fatalf("ConfigFromJSON: %v", err)
}
cl := &cloudLaunch{
Config: c,
oauthClient: jwtConf.Client(oauth2.NoContext),
}
cl.computeService, _ = compute.New(cl.oauthClient)
cl.uploadBinary()
cl.createInstance()
os.Exit(0)
}
func (c *Config) restartLoop() {
if !metadata.OnGCE() {
return
}
if c.RestartPolicy == RestartNever {
return
}
url := c.binaryURL()
var lastEtag string
for {
res, err := http.Head(url + "?" + fmt.Sprint(time.Now().Unix()))
if err != nil {
log.Printf("Warning: %v", err)
time.Sleep(15 * time.Second)
continue
}
etag := res.Header.Get("Etag")
if etag == "" {
log.Printf("Warning, no ETag in response: %v", res)
time.Sleep(15 * time.Second)
continue
}
if lastEtag != "" && etag != lastEtag {
log.Printf("Binary updated; restarting.")
// TODO: more graceful restart, letting systemd own the network connections.
// Then we can finish up requests here.
os.Exit(0)
}
lastEtag = etag
time.Sleep(15 * time.Second)
}
}
// uploadBinary uploads the currently-running Linux binary.
// It crashes if it fails.
func (cl *cloudLaunch) uploadBinary() {
ctx := context.Background()
if cl.BinaryBucket == "" {
log.Fatal("cloudlaunch: Config.BinaryBucket is empty")
}
stoClient, err := storage.NewClient(ctx, option.WithHTTPClient(cl.oauthClient))
if err != nil {
log.Fatal(err)
}
w := stoClient.Bucket(cl.BinaryBucket).Object(cl.binaryObject()).NewWriter(ctx)
if err != nil {
log.Fatal(err)
}
w.ACL = []storage.ACLRule{
// If you don't give the owners access, the web UI seems to
// have a bug and doesn't have access to see that it's public, so
// won't render the "Shared Publicly" link. So we do that, even
// though it's dumb and unnecessary otherwise:
{
Entity: storage.ACLEntity("project-owners-" + cl.GCEProjectID),
Role: storage.RoleOwner,
},
// Public, so our systemd unit can get it easily:
{
Entity: storage.AllUsers,
Role: storage.RoleReader,
},
}
w.CacheControl = "no-cache"
selfPath := getSelfPath()
log.Printf("Uploading %q to %v", selfPath, cl.binaryURL())
f, err := os.Open(selfPath)
if err != nil {
log.Fatal(err)
}
defer f.Close()
n, err := io.Copy(w, f)
if err != nil {
log.Fatal(err)
}
if err := w.Close(); err != nil {
log.Fatal(err)
}
log.Printf("Uploaded %d bytes", n)
}
func getSelfPath() string {
if runtime.GOOS != "linux" {
panic("TODO")
}
v, err := os.Readlink("/proc/self/exe")
if err != nil {
log.Fatal(err)
}
return v
}
func zoneInRegion(zone, regionURL string) bool {
if zone == "" {
panic("empty zone")
}
if regionURL == "" {
panic("empty regionURL")
}
// zone is like "us-central1-f"
// regionURL is like "https://www.googleapis.com/compute/v1/projects/camlistore-website/regions/us-central1"
region := path.Base(regionURL) // "us-central1"
if region == "" {
panic("empty region")
}
return strings.HasPrefix(zone, region)
}
// findIP finds an IP address to use, or returns the empty string if none is found.
// It tries to find a reserved one in the same region where the name of the reserved IP
// is "NAME-ip" and the IP is not in use.
func (cl *cloudLaunch) findIP() string {
// Try to find it by name.
aggAddrList, err := cl.computeService.Addresses.AggregatedList(cl.GCEProjectID).Do()
if err != nil {
log.Fatal(err)
}
// https://godoc.org/google.golang.org/api/compute/v1#AddressAggregatedList
var ip string
IPLoop:
for _, asl := range aggAddrList.Items {
for _, addr := range asl.Addresses {
log.Printf(" addr: %#v", addr)
if addr.Name == cl.Name+"-ip" && addr.Status == "RESERVED" && zoneInRegion(cl.zone(), addr.Region) {
ip = addr.Address
break IPLoop
}
}
}
return ip
}
func (cl *cloudLaunch) createInstance() {
inst := cl.lookupInstance()
if inst != nil {
log.Printf("Instance exists; not re-creating.")
return
}
log.Printf("Instance doesn't exist; creating...")
ip := cl.findIP()
log.Printf("Found IP: %v", ip)
cloudConfig := strings.NewReplacer(
"$NAME", cl.Name,
"$URL", cl.binaryURL(),
"$REBOOT", cl.updateStrategy(),
).Replace(baseConfig)
instance := &compute.Instance{
Name: cl.instName(),
Description: cl.Name,
MachineType: cl.machineTypeURL(),
Disks: []*compute.AttachedDisk{cl.instanceDisk()},
Tags: &compute.Tags{
Items: []string{"http-server", "https-server"},
},
Metadata: &compute.Metadata{
Items: []*compute.MetadataItems{
{
Key: "user-data",
Value: googleapi.String(cloudConfig),
},
},
},
NetworkInterfaces: []*compute.NetworkInterface{
&compute.NetworkInterface{
AccessConfigs: []*compute.AccessConfig{
&compute.AccessConfig{
Type: "ONE_TO_ONE_NAT",
Name: "External NAT",
NatIP: ip,
},
},
Network: cl.projectAPIURL() + "/global/networks/default",
},
},
ServiceAccounts: []*compute.ServiceAccount{
{
Email: "default",
Scopes: cl.Scopes,
},
},
}
log.Printf("Creating instance...")
op, err := cl.computeService.Instances.Insert(cl.GCEProjectID, cl.zone(), instance).Do()
if err != nil {
log.Fatalf("Failed to create instance: %v", err)
}
opName := op.Name
log.Printf("Created. Waiting on operation %v", opName)
OpLoop:
for {
time.Sleep(2 * time.Second)
op, err := cl.computeService.ZoneOperations.Get(cl.GCEProjectID, cl.zone(), opName).Do()
if err != nil {
log.Fatalf("Failed to get op %s: %v", opName, err)
}
switch op.Status {
case "PENDING", "RUNNING":
log.Printf("Waiting on operation %v", opName)
continue
case "DONE":
if op.Error != nil {
for _, operr := range op.Error.Errors {
log.Printf("Error: %+v", operr)
}
log.Fatalf("Failed to start.")
}
log.Printf("Success. %+v", op)
break OpLoop
default:
log.Fatalf("Unknown status %q: %+v", op.Status, op)
}
}
inst, err = cl.computeService.Instances.Get(cl.GCEProjectID, cl.zone(), cl.instName()).Do()
if err != nil {
log.Fatalf("Error getting instance after creation: %v", err)
}
ij, _ := json.MarshalIndent(inst, "", " ")
log.Printf("%s", ij)
log.Printf("Instance created.")
os.Exit(0)
}
// returns nil if instance doesn't exist.
func (cl *cloudLaunch) lookupInstance() *compute.Instance {
inst, err := cl.computeService.Instances.Get(cl.GCEProjectID, cl.zone(), cl.instName()).Do()
if ae, ok := err.(*googleapi.Error); ok && ae.Code == 404 {
return nil
} else if err != nil {
log.Fatalf("Instances.Get: %v", err)
}
return inst
}
func (cl *cloudLaunch) instanceDisk() *compute.AttachedDisk {
imageURL, err := gceutil.CoreOSImageURL(cl.oauthClient)
if err != nil {
log.Fatalf("error looking up latest CoreOS stable image: %v", err)
}
diskName := cl.instName() + "-coreos-stateless-pd"
var diskType string
if cl.SSD {
diskType = cl.projectAPIURL() + "/zones/" + cl.zone() + "/diskTypes/pd-ssd"
}
return &compute.AttachedDisk{
AutoDelete: true,
Boot: true,
Type: "PERSISTENT",
InitializeParams: &compute.AttachedDiskInitializeParams{
DiskName: diskName,
SourceImage: imageURL,
DiskSizeGb: 50,
DiskType: diskType,
},
}
}
================================================
FILE: cloud/google/gceutil/gceutil.go
================================================
/*
Copyright 2015 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package gceutil provides utility functions to help with instances on
// Google Compute Engine.
package gceutil // import "go4.org/cloud/google/gceutil"
import (
"encoding/json"
"errors"
"net/http"
"strings"
"time"
"google.golang.org/api/compute/v1"
)
// CoreOSImageURL returns the URL of the latest stable CoreOS image for running
// on Google Compute Engine.
func CoreOSImageURL(cl *http.Client) (string, error) {
return osImageURL(cl, false)
}
// COSImageURL returns the URL of the latest stable Container-Optimized OS image
// for running on Google Compute Engine.
func COSImageURL(cl *http.Client) (string, error) {
return osImageURL(cl, true)
}
func osImageURL(cl *http.Client, cos bool) (string, error) {
project := "coreos-cloud"
if cos {
project = "cos-cloud"
}
resp, err := cl.Get("https://www.googleapis.com/compute/v1/projects/" + project + "/global/images")
if err != nil {
return "", err
}
defer resp.Body.Close()
type osImage struct {
SelfLink string
CreationTimestamp time.Time
Name string
}
type osImageList struct {
Items []osImage
}
imageList := &osImageList{}
if err := json.NewDecoder(resp.Body).Decode(imageList); err != nil {
return "", err
}
if imageList == nil || len(imageList.Items) == 0 {
return "", errors.New("no images list in response")
}
imageURL := ""
var max time.Time // latest stable image creation time
imgPrefix := "coreos-stable"
if cos {
imgPrefix = "cos-stable"
}
for _, v := range imageList.Items {
if !strings.HasPrefix(v.Name, imgPrefix) {
continue
}
if v.CreationTimestamp.After(max) {
max = v.CreationTimestamp
imageURL = v.SelfLink
}
}
if imageURL == "" {
if cos {
return "", errors.New("no stable Container-Optimized OS image found")
}
return "", errors.New("no stable coreOS image found")
}
return imageURL, nil
}
// InstanceGroupAndManager contains both an InstanceGroup and
// its InstanceGroupManager, if any.
type InstanceGroupAndManager struct {
Group *compute.InstanceGroup
// Manager is the manager of the Group. It may be nil.
Manager *compute.InstanceGroupManager
}
// InstanceGroups returns all the instance groups in a project's zone, along
// with their associated InstanceGroupManagers.
// The returned map is keyed by the instance group identifier URL.
func InstanceGroups(svc *compute.Service, proj, zone string) (map[string]InstanceGroupAndManager, error) {
managerList, err := svc.InstanceGroupManagers.List(proj, zone).Do()
if err != nil {
return nil, err
}
if managerList.NextPageToken != "" {
return nil, errors.New("too many managers; pagination not supported")
}
managedBy := make(map[string]*compute.InstanceGroupManager) // instance group URL -> its manager
for _, it := range managerList.Items {
managedBy[it.InstanceGroup] = it
}
groupList, err := svc.InstanceGroups.List(proj, zone).Do()
if err != nil {
return nil, err
}
if groupList.NextPageToken != "" {
return nil, errors.New("too many instance groups; pagination not supported")
}
ret := make(map[string]InstanceGroupAndManager)
for _, it := range groupList.Items {
ret[it.SelfLink] = InstanceGroupAndManager{it, managedBy[it.SelfLink]}
}
return ret, nil
}
================================================
FILE: cloud/google/gcsutil/storage.go
================================================
/*
Copyright 2015 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package gcsutil provides tools for accessing Google Cloud Storage until they can be
// completely replaced by cloud.google.com/go/storage.
package gcsutil // import "go4.org/cloud/google/gcsutil"
import (
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"cloud.google.com/go/storage"
"go4.org/ctxutil"
"golang.org/x/net/context"
)
const gsAccessURL = "https://storage.googleapis.com"
// An Object holds the name of an object (its bucket and key) within
// Google Cloud Storage.
type Object struct {
Bucket string
Key string
}
func (o *Object) valid() error {
if o == nil {
return errors.New("invalid nil Object")
}
if o.Bucket == "" {
return errors.New("missing required Bucket field in Object")
}
if o.Key == "" {
return errors.New("missing required Key field in Object")
}
return nil
}
// A SizedObject holds the bucket, key, and size of an object.
type SizedObject struct {
Object
Size int64
}
func (o *Object) String() string {
if o == nil {
return "<nil *Object>"
}
return fmt.Sprintf("%v/%v", o.Bucket, o.Key)
}
func (so SizedObject) String() string {
return fmt.Sprintf("%v/%v (%vB)", so.Bucket, so.Key, so.Size)
}
// Makes a simple body-less google storage request
func simpleRequest(method, url_ string) (*http.Request, error) {
req, err := http.NewRequest(method, url_, nil)
if err != nil {
return nil, err
}
req.Header.Set("x-goog-api-version", "2")
return req, err
}
// ErrInvalidRange is used when the server has returned http.StatusRequestedRangeNotSatisfiable.
var ErrInvalidRange = errors.New("gcsutil: requested range not satisfiable")
// GetPartialObject fetches part of a Google Cloud Storage object.
// This function relies on the ctx ctxutil.HTTPClient value being set to an OAuth2
// authorized and authenticated HTTP client.
// If length is negative, the rest of the object is returned.
// It returns ErrInvalidRange if the server replies with http.StatusRequestedRangeNotSatisfiable.
// The caller must call Close on the returned value.
func GetPartialObject(ctx context.Context, obj Object, offset, length int64) (io.ReadCloser, error) {
if offset < 0 {
return nil, errors.New("invalid negative offset")
}
if err := obj.valid(); err != nil {
return nil, err
}
req, err := simpleRequest("GET", gsAccessURL+"/"+obj.Bucket+"/"+obj.Key)
if err != nil {
return nil, err
}
if length >= 0 {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-%d", offset, offset+length-1))
} else {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
req.Cancel = ctx.Done()
res, err := ctxutil.Client(ctx).Do(req)
if err != nil {
return nil, fmt.Errorf("GET (offset=%d, length=%d) failed: %v\n", offset, length, err)
}
if res.StatusCode == http.StatusNotFound {
res.Body.Close()
return nil, os.ErrNotExist
}
if !(res.StatusCode == http.StatusPartialContent || (offset == 0 && res.StatusCode == http.StatusOK)) {
res.Body.Close()
if res.StatusCode == http.StatusRequestedRangeNotSatisfiable {
return nil, ErrInvalidRange
}
return nil, fmt.Errorf("GET (offset=%d, length=%d) got failed status: %v\n", offset, length, res.Status)
}
return res.Body, nil
}
// EnumerateObjects lists the objects in a bucket.
// This function relies on the ctx oauth2.HTTPClient value being set to an OAuth2
// authorized and authenticated HTTP client.
// If after is non-empty, listing will begin with lexically greater object names.
// If limit is non-zero, the length of the list will be limited to that number.
func EnumerateObjects(ctx context.Context, bucket, after string, limit int) ([]*storage.ObjectAttrs, error) {
// Build url, with query params
var params []string
if after != "" {
params = append(params, "marker="+url.QueryEscape(after))
}
if limit > 0 {
params = append(params, fmt.Sprintf("max-keys=%v", limit))
}
query := ""
if len(params) > 0 {
query = "?" + strings.Join(params, "&")
}
req, err := simpleRequest("GET", gsAccessURL+"/"+bucket+"/"+query)
if err != nil {
return nil, err
}
req.Cancel = ctx.Done()
res, err := ctxutil.Client(ctx).Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("gcsutil: bad enumerate response code: %v", res.Status)
}
var xres struct {
Contents []SizedObject
}
if err = xml.NewDecoder(res.Body).Decode(&xres); err != nil {
return nil, err
}
objAttrs := make([]*storage.ObjectAttrs, len(xres.Contents))
for k, o := range xres.Contents {
objAttrs[k] = &storage.ObjectAttrs{
Name: o.Key,
Size: o.Size,
}
}
return objAttrs, nil
}
================================================
FILE: ctxutil/ctxutil.go
================================================
/*
Copyright 2015 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package ctxutil contains golang.org/x/net/context related utilities.
package ctxutil // import "go4.org/ctxutil"
import (
"net/http"
"golang.org/x/net/context"
"golang.org/x/oauth2"
)
// HTTPClient is the context key to use with golang.org/x/net/context's WithValue function
// to associate an *http.Client value with a context.
//
// We use the same value as the oauth2 package (which first introduced this key) rather
// than creating a new one and forcing users to possibly set two.
var HTTPClient = oauth2.HTTPClient
// Client returns the HTTP client to use for the provided context.
// If ctx is non-nil and has an associated HTTP client, that client is returned.
// Otherwise, http.DefaultClient is returned.
func Client(ctx context.Context) *http.Client {
if ctx != nil {
if hc, ok := ctx.Value(HTTPClient).(*http.Client); ok {
return hc
}
}
return http.DefaultClient
}
================================================
FILE: errorutil/highlight.go
================================================
/*
Copyright 2011 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package errorutil helps make better error messages.
package errorutil // import "go4.org/errorutil"
import (
"bufio"
"bytes"
"fmt"
"io"
"strings"
)
// HighlightBytePosition takes a reader and the location in bytes of a parse
// error (for instance, from json.SyntaxError.Offset) and returns the line, column,
// and pretty-printed context around the error with an arrow indicating the exact
// position of the syntax error.
func HighlightBytePosition(f io.Reader, pos int64) (line, col int, highlight string) {
line = 1
br := bufio.NewReader(f)
lastLine := ""
thisLine := new(bytes.Buffer)
for n := int64(0); n < pos; n++ {
b, err := br.ReadByte()
if err != nil {
break
}
if b == '\n' {
lastLine = thisLine.String()
thisLine.Reset()
line++
col = 1
} else {
col++
thisLine.WriteByte(b)
}
}
if line > 1 {
highlight += fmt.Sprintf("%5d: %s\n", line-1, lastLine)
}
highlight += fmt.Sprintf("%5d: %s\n", line, thisLine.String())
highlight += fmt.Sprintf("%s^\n", strings.Repeat(" ", col+5))
return
}
================================================
FILE: fault/fault.go
================================================
/*
Copyright 2014 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package fault handles fault injection for testing.
package fault // import "go4.org/fault"
import (
"errors"
"math/rand"
"os"
"strconv"
"strings"
)
var fakeErr = errors.New("fake injected error for testing")
// An Injector reports whether fake errors should be returned.
type Injector struct {
failPercent int
}
// NewInjector returns a new fault injector with the given name. The
// environment variable "FAULT_" + capital(name) + "_FAIL_PERCENT"
// controls the percentage of requests that fail. If undefined or
// zero, no requests fail.
func NewInjector(name string) *Injector {
var failPercent, _ = strconv.Atoi(os.Getenv("FAULT_" + strings.ToUpper(name) + "_FAIL_PERCENT"))
return &Injector{
failPercent: failPercent,
}
}
// ShouldFail reports whether a fake error should be returned.
func (in *Injector) ShouldFail() bool {
return in.failPercent > 0 && in.failPercent > rand.Intn(100)
}
// FailErr checks ShouldFail and, if true, assigns a fake error to err
// and returns true.
func (in *Injector) FailErr(err *error) bool {
if !in.ShouldFail() {
return false
}
*err = fakeErr
return true
}
================================================
FILE: go.mod
================================================
module go4.org
go 1.24.0
require (
cloud.google.com/go/compute/metadata v0.9.0
cloud.google.com/go/storage v1.56.0
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd
golang.org/x/net v0.48.0
golang.org/x/oauth2 v0.34.0
golang.org/x/sys v0.40.0
google.golang.org/api v0.259.0
)
require (
cel.dev/expr v0.24.0 // indirect
cloud.google.com/go v0.121.6 // indirect
cloud.google.com/go/auth v0.18.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/iam v1.5.3 // indirect
cloud.google.com/go/monitoring v1.24.3 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.53.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.53.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f // indirect
github.com/envoyproxy/go-control-plane/envoy v1.35.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/googleapis/gax-go/v2 v2.16.0 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.38.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/sdk v1.38.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/time v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b // indirect
google.golang.org/grpc v1.78.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
)
================================================
FILE: go.sum
================================================
cel.dev/expr v0.24.0 h1:56OvJKSH3hDGL0ml5uSxZmz3/3Pq4tJ+fb1unVLAFcY=
cel.dev/expr v0.24.0/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
cloud.google.com/go v0.121.6 h1:waZiuajrI28iAf40cWgycWNgaXPO06dupuS+sgibK6c=
cloud.google.com/go v0.121.6/go.mod h1:coChdst4Ea5vUpiALcYKXEpR1S9ZgXbhEzzMcMR66vI=
cloud.google.com/go/auth v0.18.0 h1:wnqy5hrv7p3k7cShwAU/Br3nzod7fxoqG+k0VZ+/Pk0=
cloud.google.com/go/auth v0.18.0/go.mod h1:wwkPM1AgE1f2u6dG443MiWoD8C3BtOywNsUMcUTVDRo=
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
cloud.google.com/go/iam v1.5.3 h1:+vMINPiDF2ognBJ97ABAYYwRgsaqxPbQDlMnbHMjolc=
cloud.google.com/go/iam v1.5.3/go.mod h1:MR3v9oLkZCTlaqljW6Eb2d3HGDGK5/bDv93jhfISFvU=
cloud.google.com/go/logging v1.13.1 h1:O7LvmO0kGLaHY/gq8cV7T0dyp6zJhYAOtZPX4TF3QtY=
cloud.google.com/go/logging v1.13.1/go.mod h1:XAQkfkMBxQRjQek96WLPNze7vsOmay9H5PqfsNYDqvw=
cloud.google.com/go/longrunning v0.7.0 h1:FV0+SYF1RIj59gyoWDRi45GiYUMM3K1qO51qoboQT1E=
cloud.google.com/go/longrunning v0.7.0/go.mod h1:ySn2yXmjbK9Ba0zsQqunhDkYi0+9rlXIwnoAf+h+TPY=
cloud.google.com/go/monitoring v1.24.3 h1:dde+gMNc0UhPZD1Azu6at2e79bfdztVDS5lvhOdsgaE=
cloud.google.com/go/monitoring v1.24.3/go.mod h1:nYP6W0tm3N9H/bOw8am7t62YTzZY+zUeQ+Bi6+2eonI=
cloud.google.com/go/storage v1.56.0 h1:iixmq2Fse2tqxMbWhLWC9HfBj1qdxqAmiK8/eqtsLxI=
cloud.google.com/go/storage v1.56.0/go.mod h1:Tpuj6t4NweCLzlNbw9Z9iwxEkrSem20AetIeH/shgVU=
cloud.google.com/go/trace v1.11.7 h1:kDNDX8JkaAG3R2nq1lIdkb7FCSi1rCmsEtKVsty7p+U=
cloud.google.com/go/trace v1.11.7/go.mod h1:TNn9d5V3fQVf6s4SCveVMIBS2LJUqo73GACmq/Tky0s=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0 h1:sBEjpZlNHzK1voKq9695PJSX2o5NEXl7/OL3coiIY0c=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0/go.mod h1:P4WPRUkOhJC13W//jWpyfJNDAIpvRbAUIYLX/4jtlE0=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.53.0 h1:owcC2UnmsZycprQ5RfRgjydWhuoxg71LUfyiQdijZuM=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.53.0/go.mod h1:ZPpqegjbE99EPKsu3iUWV22A04wzGPcAY/ziSIQEEgs=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.53.0 h1:4LP6hvB4I5ouTbGgWtixJhgED6xdf67twf9PoY96Tbg=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.53.0/go.mod h1:jUZ5LYlw40WMd07qxcQJD5M40aUxrfwqQX1g7zxYnrQ=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.53.0 h1:Ron4zCA/yk6U7WOBXhTJcDpsUBG9npumK6xw2auFltQ=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.53.0/go.mod h1:cSgYe11MCNYunTnRXrKiR/tHc0eoKjICUuWpNZoVCOo=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f h1:Y8xYupdHxryycyPlc9Y+bSQAYZnetRJ70VMVKm5CKI0=
github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f/go.mod h1:HlzOvOjVBOfTGSRXRyY0OiCS/3J1akRGQQpRO/7zyF4=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/envoyproxy/go-control-plane v0.13.5-0.20251024222203-75eaa193e329 h1:K+fnvUM0VZ7ZFJf0n4L/BRlnsb9pL/GuDG6FqaH+PwM=
github.com/envoyproxy/go-control-plane v0.13.5-0.20251024222203-75eaa193e329/go.mod h1:Alz8LEClvR7xKsrq3qzoc4N0guvVNSS8KmSChGYr9hs=
github.com/envoyproxy/go-control-plane/envoy v1.35.0 h1:ixjkELDE+ru6idPxcHLj8LBVc2bFP7iBytj353BoHUo=
github.com/envoyproxy/go-control-plane/envoy v1.35.0/go.mod h1:09qwbGVuSWWAyN5t/b3iyVfz5+z8QWGrzkoqm/8SbEs=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4=
github.com/envoyproxy/protoc-gen-validate v1.2.1 h1:DEo3O99U8j4hBFwbJfrz9VtgcDfUKS7KJ7spH3d86P8=
github.com/envoyproxy/protoc-gen-validate v1.2.1/go.mod h1:d/C80l/jxXLdfEIhX1W2TmLfsJ31lvEjwamM4DxlWXU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/go-jose/go-jose/v4 v4.1.3 h1:CVLmWDhDVRa6Mi/IgCgaopNosCaHz7zrMeF9MlZRkrs=
github.com/go-jose/go-jose/v4 v4.1.3/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/martian/v3 v3.3.3 h1:DIhPTQrbPkgs2yJYdXU/eNACCG5DVQjySNRNlflZ9Fc=
github.com/google/martian/v3 v3.3.3/go.mod h1:iEPrYcgCF7jA9OtScMFQyAlZZ4YXTKEtJ1E6RWzmBA0=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.7 h1:zrn2Ee/nWmHulBx5sAVrGgAa0f2/R35S4DJwfFaUPFQ=
github.com/googleapis/enterprise-certificate-proxy v0.3.7/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
github.com/googleapis/gax-go/v2 v2.16.0 h1:iHbQmKLLZrexmb0OSsNGTeSTS0HO4YvFOG8g5E4Zd0Y=
github.com/googleapis/gax-go/v2 v2.16.0/go.mod h1:o1vfQjjNZn4+dPnRdl/4ZD7S9414Y4xA+a/6Icj6l14=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
github.com/spiffe/go-spiffe/v2 v2.6.0 h1:l+DolpxNWYgruGQVV0xsfeya3CsC7m8iBzDnMpsbLuo=
github.com/spiffe/go-spiffe/v2 v2.6.0/go.mod h1:gm2SeUoMZEtpnzPNs2Csc0D/gX33k1xIx7lEzqblHEs=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/detectors/gcp v1.38.0 h1:ZoYbqX7OaA/TAikspPl3ozPI6iY6LiIY9I8cUfm+pJs=
go.opentelemetry.io/contrib/detectors/gcp v1.38.0/go.mod h1:SU+iU7nu5ud4oCb3LQOhIZ3nRLj6FNVrKgtflbaf2ts=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0 h1:q4XOmH/0opmeuJtPsbFNivyl7bCt7yRBbeEm2sC/XtQ=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0/go.mod h1:snMWehoOh2wsEwnvvwtDyFCxVeDAODenXHtn5vzrKjo=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0 h1:rixTyDGXFxRy1xzhKrotaHy3/KXdPhlWARrCgK+eqUY=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0/go.mod h1:dowW6UsM9MKbJq5JTz2AMVp3/5iW5I/TStsk8S+CfHw=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
go.opentelemetry.io/otel/sdk v1.38.0 h1:l48sr5YbNf2hpCUj/FoGhW9yDkl+Ma+LrVl8qaM5b+E=
go.opentelemetry.io/otel/sdk v1.38.0/go.mod h1:ghmNdGlVemJI3+ZB5iDEuk4bWA3GkTpW+DOoZMYBVVg=
go.opentelemetry.io/otel/sdk/metric v1.38.0 h1:aSH66iL0aZqo//xXzQLYozmWrXxyFkBJ6qT5wthqPoM=
go.opentelemetry.io/otel/sdk/metric v1.38.0/go.mod h1:dg9PBnW9XdQ1Hd6ZnRz689CbtrUp0wMMs9iPcgT9EZA=
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ=
golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
google.golang.org/api v0.259.0 h1:90TaGVIxScrh1Vn/XI2426kRpBqHwWIzVBzJsVZ5XrQ=
google.golang.org/api v0.259.0/go.mod h1:LC2ISWGWbRoyQVpxGntWwLWN/vLNxxKBK9KuJRI8Te4=
google.golang.org/genproto v0.0.0-20251202230838-ff82c1b0f217 h1:GvESR9BIyHUahIb0NcTum6itIWtdoglGX+rnGxm2934=
google.golang.org/genproto v0.0.0-20251202230838-ff82c1b0f217/go.mod h1:yJ2HH4EHEDTd3JiLmhds6NkJ17ITVYOdV3m3VKOnws0=
google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 h1:fCvbg86sFXwdrl5LgVcTEvNC+2txB5mgROGmRL5mrls=
google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217/go.mod h1:+rXWjjaukWZun3mLfjmVnQi18E1AsFbDN9QdJ5YXLto=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b h1:Mv8VFug0MP9e5vUxfBcE3vUkV6CImK3cMNMIDFjmzxU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=
google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc=
google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
================================================
FILE: go4test/cloudlaunch/serve_on_cloud.go
================================================
//go:build ignore
/*
Copyright 2016 The Go4 Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// The serve_on_cloud program deploys an HTTP server on Google Compute Engine,
// serving from Google Cloud Storage. Its purpose is to help testing
// go4.org/cloud/cloudlaunch and go4.org/wkfs/gcs.
package main
import (
"flag"
"fmt"
"log"
"net/http"
"os"
"path"
"regexp"
"time"
"go4.org/cloud/cloudlaunch"
"go4.org/wkfs"
_ "go4.org/wkfs/gcs"
"cloud.google.com/go/compute/metadata"
compute "google.golang.org/api/compute/v1"
storageapi "google.golang.org/api/storage/v1"
)
var httpAddr = flag.String("http", ":80", "HTTP address")
var gcsBucket string
func serveHTTP(w http.ResponseWriter, r *http.Request) {
rc, err := wkfs.Open(path.Join("/gcs", gcsBucket, r.URL.Path))
if err != nil {
http.Error(w, fmt.Sprintf("could not open %v: %v", r.URL.Path, err), 500)
return
}
defer rc.Close()
http.ServeContent(w, r, r.URL.Path, time.Now(), rc)
}
func main() {
if !metadata.OnGCE() {
bucket := os.Getenv("GCSBUCKET")
if bucket == "" {
log.Fatal("You need to set the GCSBUCKET env var to specify the Google Cloud Storage bucket to serve from.")
}
projectID := os.Getenv("GCEPROJECTID")
if projectID == "" {
log.Fatal("You need to set the GCEPROJECTID env var to specify the Google Cloud project where the instance will run.")
}
(&cloudlaunch.Config{
Name: "serveoncloud",
BinaryBucket: bucket,
GCEProjectID: projectID,
Scopes: []string{
storageapi.DevstorageFullControlScope,
compute.ComputeScope,
},
}).MaybeDeploy()
return
}
flag.Parse()
storageURLRxp := regexp.MustCompile(`https://storage.googleapis.com/(.+?)/serveoncloud.*`)
cloudConfig, err := metadata.InstanceAttributeValue("user-data")
if err != nil || cloudConfig == "" {
log.Fatalf("could not get cloud config from metadata: %v", err)
}
m := storageURLRxp.FindStringSubmatch(cloudConfig)
if len(m) < 2 {
log.Fatal("storage URL not found in cloud config")
}
gcsBucket = m[1]
http.HandleFunc("/", serveHTTP)
log.Fatal(http.ListenAndServe(*httpAddr, nil))
}
================================================
FILE: jsonconfig/eval.go
================================================
/*
Copyright 2011 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package jsonconfig
import (
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"path/filepath"
"regexp"
"runtime"
"strconv"
"strings"
"go4.org/errorutil"
"go4.org/wkfs"
)
type stringVector struct {
v []string
}
func (v *stringVector) Push(s string) {
v.v = append(v.v, s)
}
func (v *stringVector) Pop() {
v.v = v.v[:len(v.v)-1]
}
func (v *stringVector) Last() string {
return v.v[len(v.v)-1]
}
// A File is the type returned by ConfigParser.Open.
type File interface {
io.ReadSeeker
io.Closer
Name() string
}
// ConfigParser specifies the environment for parsing a config file
// and evaluating expressions.
type ConfigParser struct {
rootJSON Obj
touchedFiles map[string]bool
includeStack stringVector
// Open optionally specifies an opener function.
Open func(filename string) (File, error)
// IncludeDirs optionally specifies where to find the other config files which are child
// objects of this config, if any. Even if nil, the working directory is always searched
// first.
IncludeDirs []string
}
func (c *ConfigParser) open(filename string) (File, error) {
if c.Open == nil {
return wkfs.Open(filename)
}
return c.Open(filename)
}
// Validates variable names for config _env expresssions
var envPattern = regexp.MustCompile(`\$\{[A-Za-z0-9_]+\}`)
// ReadFile parses the provided path and returns the config file.
// If path is empty, the c.Open function must be defined.
func (c *ConfigParser) ReadFile(path string) (Obj, error) {
if path == "" && c.Open == nil {
return nil, errors.New("ReadFile of empty string but Open hook not defined")
}
c.touchedFiles = make(map[string]bool)
var err error
c.rootJSON, err = c.recursiveReadJSON(path)
return c.rootJSON, err
}
// Decodes and evaluates a json config file, watching for include cycles.
func (c *ConfigParser) recursiveReadJSON(configPath string) (decodedObject map[string]interface{}, err error) {
if configPath != "" {
absConfigPath, err := filepath.Abs(configPath)
if err != nil {
return nil, fmt.Errorf("Failed to expand absolute path for %s", configPath)
}
if c.touchedFiles[absConfigPath] {
return nil, fmt.Errorf("ConfigParser include cycle detected reading config: %v",
absConfigPath)
}
c.touchedFiles[absConfigPath] = true
c.includeStack.Push(absConfigPath)
defer c.includeStack.Pop()
}
var f File
if f, err = c.open(configPath); err != nil {
return nil, fmt.Errorf("Failed to open config: %v", err)
}
defer f.Close()
decodedObject = make(map[string]interface{})
dj := json.NewDecoder(f)
if err = dj.Decode(&decodedObject); err != nil {
extra := ""
if serr, ok := err.(*json.SyntaxError); ok {
if _, serr := f.Seek(0, os.SEEK_SET); serr != nil {
log.Fatalf("seek error: %v", serr)
}
line, col, highlight := errorutil.HighlightBytePosition(f, serr.Offset)
extra = fmt.Sprintf(":\nError at line %d, column %d (file offset %d):\n%s",
line, col, serr.Offset, highlight)
}
return nil, fmt.Errorf("error parsing JSON object in config file %s%s\n%v",
f.Name(), extra, err)
}
if err = c.evaluateExpressions(decodedObject, nil, false); err != nil {
return nil, fmt.Errorf("error expanding JSON config expressions in %s:\n%v",
f.Name(), err)
}
return decodedObject, nil
}
var regFunc = map[string]expanderFunc{}
// RegisterFunc registers a new function that may be called from JSON
// configs using an array of the form ["_name", arg0, argN...].
// The provided name must begin with an underscore.
func RegisterFunc(name string, fn func(c *ConfigParser, v []interface{}) (interface{}, error)) {
if len(name) < 2 || !strings.HasPrefix(name, "_") {
panic("illegal name")
}
if _, dup := regFunc[name]; dup {
panic("duplicate registration of " + name)
}
regFunc[name] = fn
}
type expanderFunc func(c *ConfigParser, v []interface{}) (interface{}, error)
func namedExpander(name string) (fn expanderFunc, ok bool) {
switch name {
case "_env":
return (*ConfigParser).expandEnv, true
case "_fileobj":
return (*ConfigParser).expandFile, true
}
fn, ok = regFunc[name]
return
}
func (c *ConfigParser) evalValue(v interface{}) (interface{}, error) {
sl, ok := v.([]interface{})
if !ok {
return v, nil
}
if name, ok := sl[0].(string); ok {
if expander, ok := namedExpander(name); ok {
newval, err := expander(c, sl[1:])
if err != nil {
return nil, err
}
return newval, nil
}
}
for i, oldval := range sl {
newval, err := c.evalValue(oldval)
if err != nil {
return nil, err
}
sl[i] = newval
}
return v, nil
}
// CheckTypes parses m and returns an error if it encounters a type or value
// that is not supported by this package.
func (c *ConfigParser) CheckTypes(m map[string]interface{}) error {
return c.evaluateExpressions(m, nil, true)
}
// evaluateExpressions parses recursively m, populating it with the values
// that are found, unless testOnly is true.
func (c *ConfigParser) evaluateExpressions(m map[string]interface{}, seenKeys []string, testOnly bool) error {
for k, ei := range m {
thisPath := append(seenKeys, k)
switch subval := ei.(type) {
case string, bool, float64, nil:
continue
case []interface{}:
if len(subval) == 0 {
continue
}
evaled, err := c.evalValue(subval)
if err != nil {
return fmt.Errorf("%s: value error %v", strings.Join(thisPath, "."), err)
}
if !testOnly {
m[k] = evaled
}
case map[string]interface{}:
if err := c.evaluateExpressions(subval, thisPath, testOnly); err != nil {
return err
}
default:
return fmt.Errorf("%s: unhandled type %T", strings.Join(thisPath, "."), ei)
}
}
return nil
}
// Permit either:
// ["_env", "VARIABLE"] (required to be set)
// or ["_env", "VARIABLE", "default_value"]
func (c *ConfigParser) expandEnv(v []interface{}) (interface{}, error) {
hasDefault := false
def := ""
if len(v) < 1 || len(v) > 2 {
return "", fmt.Errorf("_env expansion expected 1 or 2 args, got %d", len(v))
}
s, ok := v[0].(string)
if !ok {
return "", fmt.Errorf("Expected a string after _env expansion; got %#v", v[0])
}
boolDefault, wantsBool := false, false
if len(v) == 2 {
hasDefault = true
switch vdef := v[1].(type) {
case string:
def = vdef
case bool:
wantsBool = true
boolDefault = vdef
default:
return "", fmt.Errorf("Expected default value in %q _env expansion; got %#v", s, v[1])
}
}
var err error
expanded := envPattern.ReplaceAllStringFunc(s, func(match string) string {
envVar := match[2 : len(match)-1]
val := os.Getenv(envVar)
// Special case:
if val == "" && envVar == "USER" && runtime.GOOS == "windows" {
val = os.Getenv("USERNAME")
}
if val == "" {
if hasDefault {
return def
}
err = fmt.Errorf("couldn't expand environment variable %q", envVar)
}
return val
})
if wantsBool {
if expanded == "" {
return boolDefault, nil
}
return strconv.ParseBool(expanded)
}
return expanded, err
}
func (c *ConfigParser) expandFile(v []interface{}) (exp interface{}, err error) {
if len(v) != 1 {
return "", fmt.Errorf("_file expansion expected 1 arg, got %d", len(v))
}
var incPath string
if incPath, err = c.ConfigFilePath(v[0].(string)); err != nil {
return "", fmt.Errorf("Included config does not exist: %v", v[0])
}
if exp, err = c.recursiveReadJSON(incPath); err != nil {
return "", fmt.Errorf("In file included from %s:\n%v",
c.includeStack.Last(), err)
}
return exp, nil
}
// ConfigFilePath checks if configFile is found and returns a usable path to it.
// It first checks if configFile is an absolute path, or if it's found in the
// current working directory. If not, it then checks if configFile is in one of
// c.IncludeDirs. It returns an error if configFile is absolute and could not be
// statted, or os.ErrNotExist if configFile was not found.
func (c *ConfigParser) ConfigFilePath(configFile string) (path string, err error) {
// Try to open as absolute / relative to CWD
_, err = os.Stat(configFile)
if err != nil && filepath.IsAbs(configFile) {
return "", err
}
if err == nil {
return configFile, nil
}
for _, d := range c.IncludeDirs {
if _, err := os.Stat(filepath.Join(d, configFile)); err == nil {
return filepath.Join(d, configFile), nil
}
}
return "", os.ErrNotExist
}
================================================
FILE: jsonconfig/jsonconfig.go
================================================
/*
Copyright 2011 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package jsonconfig defines a helper type for JSON objects to be
// used for configuration.
package jsonconfig // import "go4.org/jsonconfig"
import (
"errors"
"fmt"
"sort"
"strconv"
"strings"
)
// Obj is a JSON configuration map.
type Obj map[string]interface{}
// ReadFile reads JSON config data from the specified open file, expanding
// all expressions. Use *ConfigParser.ReadFile instead if you
// need to set c.IncludeDirs.
func ReadFile(configPath string) (Obj, error) {
var c ConfigParser
return c.ReadFile(configPath)
}
func (jc Obj) RequiredObject(key string) Obj {
return jc.obj(key, false)
}
func (jc Obj) OptionalObject(key string) Obj {
return jc.obj(key, true)
}
func (jc Obj) obj(key string, optional bool) Obj {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if optional {
return make(Obj)
}
jc.appendError(fmt.Errorf("Missing required config key %q (object)", key))
return make(Obj)
}
m, ok := ei.(map[string]interface{})
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q to be an object, not %T", key, ei))
return make(Obj)
}
return m
}
func (jc Obj) RequiredString(key string) string {
return jc.string(key, nil)
}
func (jc Obj) OptionalString(key, def string) string {
return jc.string(key, &def)
}
func (jc Obj) string(key string, def *string) string {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if def != nil {
return *def
}
jc.appendError(fmt.Errorf("Missing required config key %q (string)", key))
return ""
}
s, ok := ei.(string)
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q to be a string", key))
return ""
}
return s
}
func (jc Obj) RequiredStringOrObject(key string) interface{} {
return jc.stringOrObject(key, true)
}
func (jc Obj) OptionalStringOrObject(key string) interface{} {
return jc.stringOrObject(key, false)
}
func (jc Obj) stringOrObject(key string, required bool) interface{} {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if !required {
return nil
}
jc.appendError(fmt.Errorf("Missing required config key %q (string or object)", key))
return ""
}
if _, ok := ei.(map[string]interface{}); ok {
return ei
}
if _, ok := ei.(string); ok {
return ei
}
jc.appendError(fmt.Errorf("Expected config key %q to be a string or object", key))
return ""
}
func (jc Obj) RequiredBool(key string) bool {
return jc.bool(key, nil)
}
func (jc Obj) OptionalBool(key string, def bool) bool {
return jc.bool(key, &def)
}
func (jc Obj) bool(key string, def *bool) bool {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if def != nil {
return *def
}
jc.appendError(fmt.Errorf("Missing required config key %q (boolean)", key))
return false
}
switch v := ei.(type) {
case bool:
return v
case string:
b, err := strconv.ParseBool(v)
if err != nil {
jc.appendError(fmt.Errorf("Config key %q has bad boolean format %q", key, v))
}
return b
default:
jc.appendError(fmt.Errorf("Expected config key %q to be a boolean", key))
return false
}
}
func (jc Obj) RequiredInt(key string) int {
return jc.int(key, nil)
}
func (jc Obj) OptionalInt(key string, def int) int {
return jc.int(key, &def)
}
func (jc Obj) int(key string, def *int) int {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if def != nil {
return *def
}
jc.appendError(fmt.Errorf("Missing required config key %q (integer)", key))
return 0
}
b, ok := ei.(float64)
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q to be a number", key))
return 0
}
return int(b)
}
func (jc Obj) RequiredInt64(key string) int64 {
return jc.int64(key, nil)
}
func (jc Obj) OptionalInt64(key string, def int64) int64 {
return jc.int64(key, &def)
}
func (jc Obj) int64(key string, def *int64) int64 {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if def != nil {
return *def
}
jc.appendError(fmt.Errorf("Missing required config key %q (integer)", key))
return 0
}
b, ok := ei.(float64)
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q to be a number", key))
return 0
}
return int64(b)
}
func (jc Obj) RequiredList(key string) []string {
return jc.requiredList(key, true)
}
func (jc Obj) OptionalList(key string) []string {
return jc.requiredList(key, false)
}
func (jc Obj) requiredList(key string, required bool) []string {
jc.noteKnownKey(key)
ei, ok := jc[key]
if !ok {
if required {
jc.appendError(fmt.Errorf("Missing required config key %q (list of strings)", key))
}
return nil
}
eil, ok := ei.([]interface{})
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q to be a list, not %T", key, ei))
return nil
}
sl := make([]string, len(eil))
for i, ei := range eil {
s, ok := ei.(string)
if !ok {
jc.appendError(fmt.Errorf("Expected config key %q index %d to be a string, not %T", key, i, ei))
return nil
}
sl[i] = s
}
return sl
}
func (jc Obj) noteKnownKey(key string) {
_, ok := jc["_knownkeys"]
if !ok {
jc["_knownkeys"] = make(map[string]bool)
}
jc["_knownkeys"].(map[string]bool)[key] = true
}
func (jc Obj) appendError(err error) {
ei, ok := jc["_errors"]
if ok {
jc["_errors"] = append(ei.([]error), err)
} else {
jc["_errors"] = []error{err}
}
}
// UnknownKeys returns the keys from the config that have not yet been discovered by one of the RequiredT or OptionalT calls.
func (jc Obj) UnknownKeys() []string {
ei, ok := jc["_knownkeys"]
var known map[string]bool
if ok {
known = ei.(map[string]bool)
}
var unknown []string
for k, _ := range jc {
if ok && known[k] {
continue
}
if strings.HasPrefix(k, "_") {
// Permit keys with a leading underscore as a
// form of comments.
continue
}
unknown = append(unknown, k)
}
sort.Strings(unknown)
return unknown
}
func (jc Obj) Validate() error {
unknown := jc.UnknownKeys()
for _, k := range unknown {
jc.appendError(fmt.Errorf("Unknown key %q", k))
}
ei, ok := jc["_errors"]
if !ok {
return nil
}
errList := ei.([]error)
return errors.Join(errList...)
}
================================================
FILE: jsonconfig/jsonconfig_test.go
================================================
/*
Copyright 2011 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package jsonconfig
import (
"os"
"reflect"
"strings"
"testing"
)
func testIncludes(configFile string, t *testing.T) {
var c ConfigParser
c.IncludeDirs = []string{"testdata"}
obj, err := c.ReadFile(configFile)
if err != nil {
t.Fatal(err)
}
two := obj.RequiredObject("two")
if err := obj.Validate(); err != nil {
t.Error(err)
}
if g, e := two.RequiredString("key"), "value"; g != e {
t.Errorf("sub object key = %q; want %q", g, e)
}
}
func TestIncludesCWD(t *testing.T) {
testIncludes("testdata/include1.json", t)
}
func TestIncludesIncludeDirs(t *testing.T) {
testIncludes("testdata/include1bis.json", t)
}
func TestIncludeLoop(t *testing.T) {
_, err := ReadFile("testdata/loop1.json")
if err == nil {
t.Fatal("expected an error about import cycles.")
}
if !strings.Contains(err.Error(), "include cycle detected") {
t.Fatalf("expected an error about import cycles; got: %v", err)
}
}
func TestBoolEnvs(t *testing.T) {
os.Setenv("TEST_EMPTY", "")
os.Setenv("TEST_TRUE", "true")
os.Setenv("TEST_ONE", "1")
os.Setenv("TEST_ZERO", "0")
os.Setenv("TEST_FALSE", "false")
obj, err := ReadFile("testdata/boolenv.json")
if err != nil {
t.Fatal(err)
}
if str := obj.RequiredString("emptystr"); str != "" {
t.Errorf("str = %q, want empty", str)
}
tests := []struct {
key string
want bool
}{
{"def_false", false},
{"def_true", true},
{"set_true_def_false", true},
{"set_false_def_true", false},
{"lit_true", true},
{"lit_false", false},
{"one", true},
{"zero", false},
}
for _, tt := range tests {
if v := obj.RequiredBool(tt.key); v != tt.want {
t.Errorf("key %q = %v; want %v", tt.key, v, tt.want)
}
}
if err := obj.Validate(); err != nil {
t.Error(err)
}
}
func TestListExpansion(t *testing.T) {
os.Setenv("TEST_BAR", "bar")
obj, err := ReadFile("testdata/listexpand.json")
if err != nil {
t.Fatal(err)
}
s := obj.RequiredString("str")
l := obj.RequiredList("list")
if err := obj.Validate(); err != nil {
t.Error(err)
}
want := []string{"foo", "bar"}
if !reflect.DeepEqual(l, want) {
t.Errorf("got = %#v\nwant = %#v", l, want)
}
if s != "bar" {
t.Errorf("str = %q, want %q", s, "bar")
}
}
================================================
FILE: jsonconfig/testdata/boolenv.json
================================================
{
"emptystr": ["_env", "${TEST_EMPTY}", ""],
"def_false": ["_env", "${TEST_EMPTY}", false],
"def_true": ["_env", "${TEST_EMPTY}", true],
"set_true_def_false": ["_env", "${TEST_TRUE}", false],
"set_false_def_true": ["_env", "${TEST_FALSE}", true],
"one": ["_env", "${TEST_ONE}"],
"zero": ["_env", "${TEST_ZERO}"],
"lit_true": true,
"lit_false": false
}
================================================
FILE: jsonconfig/testdata/include1.json
================================================
{
"two": ["_fileobj", "testdata/include2.json"]
}
================================================
FILE: jsonconfig/testdata/include1bis.json
================================================
{
"two": ["_fileobj", "include2.json"]
}
================================================
FILE: jsonconfig/testdata/include2.json
================================================
{
"key": "value"
}
================================================
FILE: jsonconfig/testdata/listexpand.json
================================================
{
"list": ["foo", ["_env", "${TEST_BAR}"]],
"str": ["_env", "${TEST_BAR}"]
}
================================================
FILE: jsonconfig/testdata/loop1.json
================================================
{
"obj": ["_fileobj", "testdata/loop2.json"]
}
================================================
FILE: jsonconfig/testdata/loop2.json
================================================
{
"obj": ["_fileobj", "testdata/loop1.json"]
}
================================================
FILE: legal/legal.go
================================================
/*
Copyright 2014 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package legal provides in-process storage for compiled-in licenses.
package legal // import "go4.org/legal"
var licenses []string
// RegisterLicense stores the license text.
// It doesn't check whether the text was already present.
func RegisterLicense(text string) {
licenses = append(licenses, text)
return
}
// Licenses returns a slice of the licenses.
func Licenses() []string {
return licenses
}
================================================
FILE: legal/legal_test.go
================================================
/*
Copyright 2014 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package legal
import (
"testing"
)
func TestRegisterLicense(t *testing.T) {
initial := len(licenses)
RegisterLicense("dummy")
if initial+1 != len(licenses) {
t.Fatal("didn't add a license")
}
}
================================================
FILE: lock/.gitignore
================================================
*~
================================================
FILE: lock/lock.go
================================================
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package lock is a file locking library.
package lock // import "go4.org/lock"
import (
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"sync"
)
// Lock locks the given file, creating the file if necessary. If the
// file already exists, it must have zero size or an error is returned.
// The lock is an exclusive lock (a write lock), but locked files
// should neither be read from nor written to. Such files should have
// zero size and only exist to co-ordinate ownership across processes.
//
// A nil Closer is returned if an error occurred. Otherwise, close that
// Closer to release the lock.
//
// On Linux, FreeBSD and OSX, a lock has the same semantics as fcntl(2)'s
// advisory locks. In particular, closing any other file descriptor for the
// same file will release the lock prematurely.
//
// Attempting to lock a file that is already locked by the current process
// has undefined behavior.
//
// On other operating systems, lock will fallback to using the presence and
// content of a file named name + '.lock' to implement locking behavior.
func Lock(name string) (io.Closer, error) {
abs, err := filepath.Abs(name)
if err != nil {
return nil, err
}
lockmu.Lock()
defer lockmu.Unlock()
if locked[abs] {
return nil, fmt.Errorf("file %q already locked", abs)
}
c, err := lockFn(abs)
if err != nil {
return nil, fmt.Errorf("cannot acquire lock: %v", err)
}
locked[abs] = true
return c, nil
}
var lockFn = lockPortable
// lockPortable is a portable version not using fcntl. Doesn't handle crashes as gracefully,
// since it can leave stale lock files.
func lockPortable(name string) (io.Closer, error) {
fi, err := os.Stat(name)
if err == nil && fi.Size() > 0 {
st := portableLockStatus(name)
switch st {
case statusLocked:
return nil, fmt.Errorf("file %q already locked", name)
case statusStale:
os.Remove(name)
case statusInvalid:
return nil, fmt.Errorf("can't Lock file %q: has invalid contents", name)
}
}
f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_TRUNC|os.O_EXCL, 0666)
if err != nil {
return nil, fmt.Errorf("failed to create lock file %s %v", name, err)
}
if err := json.NewEncoder(f).Encode(&pidLockMeta{OwnerPID: os.Getpid()}); err != nil {
return nil, fmt.Errorf("cannot write owner pid: %v", err)
}
return &unlocker{
f: f,
abs: name,
portable: true,
}, nil
}
type lockStatus int
const (
statusInvalid lockStatus = iota
statusLocked
statusUnlocked
statusStale
)
type pidLockMeta struct {
OwnerPID int
}
func portableLockStatus(path string) lockStatus {
f, err := os.Open(path)
if err != nil {
return statusUnlocked
}
defer f.Close()
var meta pidLockMeta
if json.NewDecoder(f).Decode(&meta) != nil {
return statusInvalid
}
if meta.OwnerPID == 0 {
return statusInvalid
}
p, err := os.FindProcess(meta.OwnerPID)
if err != nil {
// e.g. on Windows
return statusStale
}
// On unix, os.FindProcess always is true, so we have to send
// it a signal to see if it's alive.
if signalZero != nil {
if p.Signal(signalZero) != nil {
return statusStale
}
}
return statusLocked
}
var signalZero os.Signal // nil or set by lock_sigzero.go
var (
lockmu sync.Mutex
locked = map[string]bool{} // abs path -> true
)
type unlocker struct {
portable bool
f *os.File
abs string
// once guards the close method call.
once sync.Once
// err holds the error returned by Close.
err error
}
func (u *unlocker) Close() error {
u.once.Do(u.close)
return u.err
}
func (u *unlocker) close() {
lockmu.Lock()
defer lockmu.Unlock()
delete(locked, u.abs)
if u.portable {
// In the portable lock implementation, it's
// important to close before removing because
// Windows won't allow us to remove an open
// file.
if err := u.f.Close(); err != nil {
u.err = err
}
if err := os.Remove(u.abs); err != nil {
// Note that if both Close and Remove fail,
// we care more about the latter than the former
// so we'll return that error.
u.err = err
}
return
}
// In other implementatioons, it's nice for us to clean up.
// If we do do this, though, it needs to be before the
// u.f.Close below.
os.Remove(u.abs)
u.err = u.f.Close()
}
================================================
FILE: lock/lock_plan9.go
================================================
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package lock
import (
"fmt"
"io"
"os"
)
func init() {
lockFn = lockPlan9
}
func lockPlan9(name string) (io.Closer, error) {
fi, err := os.Stat(name)
if err == nil && fi.Size() > 0 {
return nil, fmt.Errorf("can't Lock file %q: has non-zero size", name)
}
f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE, os.ModeExclusive|0644)
if err != nil {
return nil, fmt.Errorf("Lock Create of %s failed: %v", name, err)
}
return &unlocker{f: f, abs: name}, nil
}
================================================
FILE: lock/lock_sigzero.go
================================================
//go:build linux || darwin || freebsd || openbsd || netbsd || dragonfly || solaris
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package lock
import "syscall"
func init() {
signalZero = syscall.Signal(0)
}
================================================
FILE: lock/lock_test.go
================================================
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package lock
import (
"bufio"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"strconv"
"testing"
)
func TestLock(t *testing.T) {
testLock(t, false)
}
func TestLockPortable(t *testing.T) {
testLock(t, true)
}
func TestLockInChild(t *testing.T) {
f := os.Getenv("TEST_LOCK_FILE")
if f == "" {
// not child
return
}
lock := Lock
if v, _ := strconv.ParseBool(os.Getenv("TEST_LOCK_PORTABLE")); v {
lock = lockPortable
}
var lk io.Closer
for scan := bufio.NewScanner(os.Stdin); scan.Scan(); {
var err error
switch scan.Text() {
case "lock":
lk, err = lock(f)
case "unlock":
err = lk.Close()
lk = nil
case "exit":
// Simulate a crash, or at least not unlocking the lock.
os.Exit(0)
default:
err = fmt.Errorf("unexpected child command %q", scan.Text())
}
if err != nil {
fmt.Println(err)
} else {
fmt.Println("")
}
}
}
func testLock(t *testing.T, portable bool) {
lock := Lock
if portable {
lock = lockPortable
}
t.Logf("test lock, portable %v", portable)
td, err := ioutil.TempDir("", "")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(td)
path := filepath.Join(td, "foo.lock")
proc := newChildProc(t, path, portable)
defer proc.kill()
t.Logf("First lock in child")
if err := proc.do("lock"); err != nil {
t.Fatalf("first lock in child process: %v", err)
}
t.Logf("Crash child")
if err := proc.do("exit"); err != nil {
t.Fatalf("crash in child process: %v", err)
}
proc = newChildProc(t, path, portable)
defer proc.kill()
t.Logf("Locking+unlocking in child...")
if err := proc.do("lock"); err != nil {
t.Fatalf("lock in child process after crashing child: %v", err)
}
if err := proc.do("unlock"); err != nil {
t.Fatalf("lock in child process after crashing child: %v", err)
}
t.Logf("Locking in parent...")
lk1, err := lock(path)
if err != nil {
t.Fatal(err)
}
t.Logf("Again in parent...")
_, err = lock(path)
if err == nil {
t.Fatal("expected second lock to fail")
}
t.Logf("Locking in child...")
if err := proc.do("lock"); err == nil {
t.Fatalf("expected lock in child process to fail")
}
t.Logf("Unlocking lock in parent")
if err := lk1.Close(); err != nil {
t.Fatal(err)
}
t.Logf("Trying lock again in child...")
if err := proc.do("lock"); err != nil {
t.Fatal(err)
}
if err := proc.do("unlock"); err != nil {
t.Fatal(err)
}
lk3, err := lock(path)
if err != nil {
t.Fatal(err)
}
lk3.Close()
}
type childLockCmd struct {
op string
reply chan<- error
}
type childProc struct {
proc *os.Process
c chan childLockCmd
}
func (c *childProc) kill() {
c.proc.Kill()
}
func (c *childProc) do(op string) error {
reply := make(chan error)
c.c <- childLockCmd{
op: op,
reply: reply,
}
return <-reply
}
func newChildProc(t *testing.T, path string, portable bool) *childProc {
cmd := exec.Command(os.Args[0], "-test.run=LockInChild$")
cmd.Env = []string{"TEST_LOCK_FILE=" + path}
toChild, err := cmd.StdinPipe()
if err != nil {
t.Fatalf("cannot make pipe: %v", err)
}
fromChild, err := cmd.StdoutPipe()
if err != nil {
t.Fatalf("cannot make pipe: %v", err)
}
cmd.Stderr = os.Stderr
if portable {
cmd.Env = append(cmd.Env, "TEST_LOCK_PORTABLE=1")
}
if err := cmd.Start(); err != nil {
t.Fatalf("cannot start child: %v", err)
}
cmdChan := make(chan childLockCmd)
go func() {
defer fromChild.Close()
defer toChild.Close()
inScan := bufio.NewScanner(fromChild)
for c := range cmdChan {
fmt.Fprintln(toChild, c.op)
ok := inScan.Scan()
if c.op == "exit" {
if ok {
c.reply <- errors.New("child did not exit")
} else {
cmd.Wait()
c.reply <- nil
}
break
}
if !ok {
panic("child exited early")
}
if errText := inScan.Text(); errText != "" {
c.reply <- errors.New(errText)
} else {
c.reply <- nil
}
}
}()
return &childProc{
c: cmdChan,
proc: cmd.Process,
}
}
================================================
FILE: lock/lock_unix.go
================================================
//go:build linux || darwin || freebsd || openbsd || netbsd || dragonfly || solaris
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package lock
import (
"fmt"
"io"
"os"
"golang.org/x/sys/unix"
)
func init() {
lockFn = lockFcntl
}
func lockFcntl(name string) (io.Closer, error) {
fi, err := os.Stat(name)
if err == nil && fi.Size() > 0 {
return nil, fmt.Errorf("can't Lock file %q: has non-zero size", name)
}
f, err := os.Create(name)
if err != nil {
return nil, fmt.Errorf("Lock Create of %s failed: %v", name, err)
}
err = unix.FcntlFlock(f.Fd(), unix.F_SETLK, &unix.Flock_t{
Type: unix.F_WRLCK,
Whence: int16(os.SEEK_SET),
Start: 0,
Len: 0, // 0 means to lock the entire file.
Pid: 0, // only used by F_GETLK
})
if err != nil {
f.Close()
return nil, fmt.Errorf("Lock FcntlFlock of %s failed: %v", name, err)
}
return &unlocker{f: f, abs: name}, nil
}
================================================
FILE: lock/lock_windows.go
================================================
/*
Copyright 2013 The Go Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package lock
import (
"fmt"
"io"
"os"
"sync"
"golang.org/x/sys/windows"
)
func init() {
lockFn = lockWindows
}
type winUnlocker struct {
h windows.Handle
abs string
// err holds the error returned by Close.
err error
// once guards the close method call.
once sync.Once
}
func (u *winUnlocker) Close() error {
u.once.Do(u.close)
return u.err
}
func (u *winUnlocker) close() {
lockmu.Lock()
defer lockmu.Unlock()
delete(locked, u.abs)
u.err = windows.CloseHandle(u.h)
}
func lockWindows(name string) (io.Closer, error) {
fi, err := os.Stat(name)
if err == nil && fi.Size() > 0 {
return nil, fmt.Errorf("can't lock file %q: %s", name, "has non-zero size")
}
handle, err := winCreateEphemeral(name)
if err != nil {
return nil, fmt.Errorf("creation of lock %s failed: %v", name, err)
}
return &winUnlocker{h: handle, abs: name}, nil
}
func winCreateEphemeral(name string) (windows.Handle, error) {
const (
FILE_ATTRIBUTE_TEMPORARY = 0x100
FILE_FLAG_DELETE_ON_CLOSE = 0x04000000
)
handle, err := windows.CreateFile(windows.StringToUTF16Ptr(name), 0, 0, nil, windows.OPEN_ALWAYS, FILE_ATTRIBUTE_TEMPORARY|FILE_FLAG_DELETE_ON_CLOSE, 0)
if err != nil {
return 0, err
}
return handle, nil
}
================================================
FILE: media/heif/bmff/bmff.go
================================================
/*
Copyright 2018 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package bmff reads ISO BMFF boxes, as used by HEIF, etc.
//
// This is not so much as a generic BMFF reader as it is a BMFF reader
// as needed by HEIF, though that may change in time. For now, only
// boxes necessary for the go4.org/media/heif package have explicit
// parsers.
//
// This package makes no API compatibility promises; it exists
// primarily for use by the go4.org/media/heif package.
package bmff
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"io/ioutil"
"strings"
)
func NewReader(r io.Reader) *Reader {
br, ok := r.(*bufio.Reader)
if !ok {
br = bufio.NewReader(r)
}
return &Reader{br: bufReader{Reader: br}}
}
type Reader struct {
br bufReader
lastBox Box // or nil
noMoreBoxes bool // a box with size 0 (the final box) was seen
}
type BoxType [4]byte
// Common box types.
var (
TypeFtyp = BoxType{'f', 't', 'y', 'p'}
TypeMeta = BoxType{'m', 'e', 't', 'a'}
)
func (t BoxType) String() string { return string(t[:]) }
func (t BoxType) EqualString(s string) bool {
// Could be cleaner, but see ohttps://github.com/golang/go/issues/24765
return len(s) == 4 && s[0] == t[0] && s[1] == t[1] && s[2] == t[2] && s[3] == t[3]
}
type parseFunc func(b box, br *bufio.Reader) (Box, error)
// Box represents a BMFF box.
type Box interface {
Size() int64 // 0 means unknown (will read to end of file)
Type() BoxType
// Parses parses the box, populating the fields
// in the returned concrete type.
//
// If Parse has already been called, Parse returns nil.
// If the box type is unknown, the returned error is ErrUnknownBox
// and it's guaranteed that no bytes have been read from the box.
Parse() (Box, error)
// Body returns the inner bytes of the box, ignoring the header.
// The body may start with the 4 byte header of a "Full Box" if the
// box's type derives from a full box. Most users will use Parse
// instead.
// Body will return a new reader at the beginning of the box if the
// outer box has already been parsed.
Body() io.Reader
}
// ErrUnknownBox is returned by Box.Parse for unrecognized box types.
var ErrUnknownBox = errors.New("heif: unknown box")
type parserFunc func(b *box, br *bufReader) (Box, error)
func boxType(s string) BoxType {
if len(s) != 4 {
panic("bogus boxType length")
}
return BoxType{s[0], s[1], s[2], s[3]}
}
var parsers = map[BoxType]parserFunc{
boxType("dinf"): parseDataInformationBox,
boxType("dref"): parseDataReferenceBox,
boxType("ftyp"): parseFileTypeBox,
boxType("hdlr"): parseHandlerBox,
boxType("iinf"): parseItemInfoBox,
boxType("infe"): parseItemInfoEntry,
boxType("iloc"): parseItemLocationBox,
boxType("ipco"): parseItemPropertyContainerBox,
boxType("ipma"): parseItemPropertyAssociation,
boxType("iprp"): parseItemPropertiesBox,
boxType("irot"): parseImageRotation,
boxType("ispe"): parseImageSpatialExtentsProperty,
boxType("meta"): parseMetaBox,
boxType("pitm"): parsePrimaryItemBox,
}
type box struct {
size int64 // 0 means unknown, will read to end of file (box container)
boxType BoxType
body io.Reader
parsed Box // if non-nil, the Parsed result
slurp []byte // if non-nil, the contents slurped to memory
}
func (b *box) Size() int64 { return b.size }
func (b *box) Type() BoxType { return b.boxType }
func (b *box) Body() io.Reader {
if b.slurp != nil {
return bytes.NewReader(b.slurp)
}
return b.body
}
func (b *box) Parse() (Box, error) {
if b.parsed != nil {
return b.parsed, nil
}
parser, ok := parsers[b.Type()]
if !ok {
return nil, ErrUnknownBox
}
v, err := parser(b, &bufReader{Reader: bufio.NewReader(b.Body())})
if err != nil {
return nil, err
}
b.parsed = v
return v, nil
}
type FullBox struct {
*box
Version uint8
Flags uint32 // 24 bits
}
// ReadBox reads the next box.
//
// If the previously read box was not read to completion, ReadBox consumes
// the rest of its data.
//
// At the end, the error is io.EOF.
func (r *Reader) ReadBox() (Box, error) {
if r.noMoreBoxes {
return nil, io.EOF
}
if r.lastBox != nil {
if _, err := io.Copy(ioutil.Discard, r.lastBox.Body()); err != nil {
return nil, err
}
}
var buf [8]byte
_, err := io.ReadFull(r.br, buf[:4])
if err != nil {
return nil, err
}
box := &box{
size: int64(binary.BigEndian.Uint32(buf[:4])),
}
_, err = io.ReadFull(r.br, box.boxType[:]) // 4 more bytes
if err != nil {
return nil, err
}
// Special cases for size:
var remain int64
switch box.size {
case 1:
// 1 means it's actually a 64-bit size, after the type.
_, err = io.ReadFull(r.br, buf[:8])
if err != nil {
return nil, err
}
box.size = int64(binary.BigEndian.Uint64(buf[:8]))
if box.size < 0 {
// Go uses int64 for sizes typically, but BMFF uses uint64.
// We assume for now that nobody actually uses boxes larger
// than int64.
return nil, fmt.Errorf("unexpectedly large box %q", box.boxType)
}
remain = box.size - 2*4 - 8
case 0:
// 0 means unknown & to read to end of file. No more boxes.
r.noMoreBoxes = true
default:
remain = box.size - 2*4
}
if remain < 0 {
return nil, fmt.Errorf("Box header for %q has size %d, suggesting %d (negative) bytes remain", box.boxType, box.size, remain)
}
if box.size > 0 {
box.body = io.LimitReader(r.br, remain)
} else {
box.body = r.br
}
r.lastBox = box
return box, nil
}
// ReadAndParseBox wraps the ReadBox method, ensuring that the read box is of type typ
// and parses successfully. It returns the parsed box.
func (r *Reader) ReadAndParseBox(typ BoxType) (Box, error) {
box, err := r.ReadBox()
if err != nil {
return nil, fmt.Errorf("error reading %q box: %v", typ, err)
}
if box.Type() != typ {
return nil, fmt.Errorf("error reading %q box: got box type %q instead", typ, box.Type())
}
pbox, err := box.Parse()
if err != nil {
return nil, fmt.Errorf("error parsing read %q box: %v", typ, err)
}
return pbox, nil
}
func readFullBox(outer *box, br *bufReader) (fb FullBox, err error) {
fb.box = outer
// Parse FullBox header.
buf, err := br.Peek(4)
if err != nil {
return FullBox{}, fmt.Errorf("failed to read 4 bytes of FullBox: %v", err)
}
fb.Version = buf[0]
buf[0] = 0
fb.Flags = binary.BigEndian.Uint32(buf[:4])
br.Discard(4)
return fb, nil
}
type FileTypeBox struct {
*box
MajorBrand string // 4 bytes
MinorVersion string // 4 bytes
Compatible []string // all 4 bytes
}
func parseFileTypeBox(outer *box, br *bufReader) (Box, error) {
buf, err := br.Peek(8)
if err != nil {
return nil, err
}
ft := &FileTypeBox{
box: outer,
MajorBrand: string(buf[:4]),
MinorVersion: string(buf[4:8]),
}
br.Discard(8)
for {
buf, err := br.Peek(4)
if err == io.EOF {
return ft, nil
}
if err != nil {
return nil, err
}
ft.Compatible = append(ft.Compatible, string(buf[:4]))
br.Discard(4)
}
}
type MetaBox struct {
FullBox
Children []Box
}
func parseMetaBox(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
mb := &MetaBox{FullBox: fb}
return mb, br.parseAppendBoxes(&mb.Children)
}
func (br *bufReader) parseAppendBoxes(dst *[]Box) error {
if br.err != nil {
return br.err
}
boxr := NewReader(br.Reader)
for {
inner, err := boxr.ReadBox()
if err == io.EOF {
return nil
}
if err != nil {
br.err = err
return err
}
slurp, err := ioutil.ReadAll(inner.Body())
if err != nil {
br.err = err
return err
}
inner.(*box).slurp = slurp
*dst = append(*dst, inner)
}
}
// ItemInfoEntry represents an "infe" box.
//
// TODO: currently only parses Version 2 boxes.
type ItemInfoEntry struct {
FullBox
ItemID uint16
ProtectionIndex uint16
ItemType string // always 4 bytes
Name string
// If Type == "mime":
ContentType string
ContentEncoding string
// If Type == "uri ":
ItemURIType string
}
func parseItemInfoEntry(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
ie := &ItemInfoEntry{FullBox: fb}
if fb.Version != 2 {
return nil, fmt.Errorf("TODO: found version %d infe box. Only 2 is supported now.", fb.Version)
}
ie.ItemID, _ = br.readUint16()
ie.ProtectionIndex, _ = br.readUint16()
if !br.ok() {
return nil, br.err
}
buf, err := br.Peek(4)
if err != nil {
return nil, err
}
ie.ItemType = string(buf[:4])
ie.Name, _ = br.readString()
switch ie.ItemType {
case "mime":
ie.ContentType, _ = br.readString()
if br.anyRemain() {
ie.ContentEncoding, _ = br.readString()
}
case "uri ":
ie.ItemURIType, _ = br.readString()
}
if !br.ok() {
return nil, br.err
}
return ie, nil
}
// ItemInfoBox represents an "iinf" box.
type ItemInfoBox struct {
FullBox
Count uint16
ItemInfos []*ItemInfoEntry
}
func parseItemInfoBox(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
ib := &ItemInfoBox{FullBox: fb}
ib.Count, _ = br.readUint16()
var itemInfos []Box
br.parseAppendBoxes(&itemInfos)
if br.ok() {
for _, box := range itemInfos {
pb, err := box.Parse()
if err != nil {
return nil, fmt.Errorf("error parsing ItemInfoEntry in ItemInfoBox: %v", err)
}
if iie, ok := pb.(*ItemInfoEntry); ok {
ib.ItemInfos = append(ib.ItemInfos, iie)
}
}
}
if !br.ok() {
return FullBox{}, br.err
}
return ib, nil
}
// bufReader adds some HEIF/BMFF-specific methods around a *bufio.Reader.
type bufReader struct {
*bufio.Reader
err error // sticky error
}
// ok reports whether all previous reads have been error-free.
func (br *bufReader) ok() bool { return br.err == nil }
func (br *bufReader) anyRemain() bool {
if br.err != nil {
return false
}
_, err := br.Peek(1)
return err == nil
}
func (br *bufReader) readUintN(bits uint8) (uint64, error) {
if br.err != nil {
return 0, br.err
}
if bits == 0 {
return 0, nil
}
nbyte := bits / 8
buf, err := br.Peek(int(nbyte))
if err != nil {
br.err = err
return 0, err
}
defer br.Discard(int(nbyte))
switch bits {
case 8:
return uint64(buf[0]), nil
case 16:
return uint64(binary.BigEndian.Uint16(buf[:2])), nil
case 32:
return uint64(binary.BigEndian.Uint32(buf[:4])), nil
case 64:
return binary.BigEndian.Uint64(buf[:8]), nil
default:
br.err = fmt.Errorf("invalid uintn read size")
return 0, br.err
}
}
func (br *bufReader) readUint8() (uint8, error) {
if br.err != nil {
return 0, br.err
}
v, err := br.ReadByte()
if err != nil {
br.err = err
return 0, err
}
return v, nil
}
func (br *bufReader) readUint16() (uint16, error) {
if br.err != nil {
return 0, br.err
}
buf, err := br.Peek(2)
if err != nil {
br.err = err
return 0, err
}
v := binary.BigEndian.Uint16(buf[:2])
br.Discard(2)
return v, nil
}
func (br *bufReader) readUint32() (uint32, error) {
if br.err != nil {
return 0, br.err
}
buf, err := br.Peek(4)
if err != nil {
br.err = err
return 0, err
}
v := binary.BigEndian.Uint32(buf[:4])
br.Discard(4)
return v, nil
}
func (br *bufReader) readString() (string, error) {
if br.err != nil {
return "", br.err
}
s0, err := br.ReadString(0)
if err != nil {
br.err = err
return "", err
}
s := strings.TrimSuffix(s0, "\x00")
if len(s) == len(s0) {
err = fmt.Errorf("unexpected non-null terminated string")
br.err = err
return "", err
}
return s, nil
}
// HEIF: ipco
type ItemPropertyContainerBox struct {
*box
Properties []Box // of ItemProperty or ItemFullProperty
}
func parseItemPropertyContainerBox(outer *box, br *bufReader) (Box, error) {
ipc := &ItemPropertyContainerBox{box: outer}
return ipc, br.parseAppendBoxes(&ipc.Properties)
}
// HEIF: iprp
type ItemPropertiesBox struct {
*box
PropertyContainer *ItemPropertyContainerBox
Associations []*ItemPropertyAssociation // at least 1
}
func parseItemPropertiesBox(outer *box, br *bufReader) (Box, error) {
ip := &ItemPropertiesBox{
box: outer,
}
var boxes []Box
err := br.parseAppendBoxes(&boxes)
if err != nil {
return nil, err
}
if len(boxes) < 2 {
return nil, fmt.Errorf("expect at least 2 boxes in children; got 0")
}
cb, err := boxes[0].Parse()
if err != nil {
return nil, fmt.Errorf("failed to parse first box, %q: %v", boxes[0].Type(), err)
}
var ok bool
ip.PropertyContainer, ok = cb.(*ItemPropertyContainerBox)
if !ok {
return nil, fmt.Errorf("unexpected type %T for ItemPropertieBox.PropertyContainer", cb)
}
// Association boxes
ip.Associations = make([]*ItemPropertyAssociation, 0, len(boxes)-1)
for _, box := range boxes[1:] {
boxp, err := box.Parse()
if err != nil {
return nil, fmt.Errorf("failed to parse association box: %v", err)
}
ipa, ok := boxp.(*ItemPropertyAssociation)
if !ok {
return nil, fmt.Errorf("unexpected box %q instead of ItemPropertyAssociation", boxp.Type())
}
ip.Associations = append(ip.Associations, ipa)
}
return ip, nil
}
type ItemPropertyAssociation struct {
FullBox
EntryCount uint32
Entries []ItemPropertyAssociationItem
}
// not a box
type ItemProperty struct {
Essential bool
Index uint16
}
// not a box
type ItemPropertyAssociationItem struct {
ItemID uint32
AssociationsCount int // as declared
Associations []ItemProperty // as parsed
}
func parseItemPropertyAssociation(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
ipa := &ItemPropertyAssociation{FullBox: fb}
count, _ := br.readUint32()
ipa.EntryCount = count
for i := uint64(0); i < uint64(count) && br.ok(); i++ {
var itemID uint32
if fb.Version < 1 {
itemID16, _ := br.readUint16()
itemID = uint32(itemID16)
} else {
itemID, _ = br.readUint32()
}
assocCount, _ := br.readUint8()
ipai := ItemPropertyAssociationItem{
ItemID: itemID,
AssociationsCount: int(assocCount),
}
for j := 0; j < int(assocCount) && br.ok(); j++ {
first, _ := br.readUint8()
essential := first&(1<<7) != 0
first &^= byte(1 << 7)
var index uint16
if fb.Flags&1 != 0 {
second, _ := br.readUint8()
index = uint16(first)<<8 | uint16(second)
} else {
index = uint16(first)
}
ipai.Associations = append(ipai.Associations, ItemProperty{
Essential: essential,
Index: index,
})
}
ipa.Entries = append(ipa.Entries, ipai)
}
if !br.ok() {
return nil, br.err
}
return ipa, nil
}
type ImageSpatialExtentsProperty struct {
FullBox
ImageWidth uint32
ImageHeight uint32
}
func parseImageSpatialExtentsProperty(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
w, err := br.readUint32()
if err != nil {
return nil, err
}
h, err := br.readUint32()
if err != nil {
return nil, err
}
return &ImageSpatialExtentsProperty{
FullBox: fb,
ImageWidth: w,
ImageHeight: h,
}, nil
}
type OffsetLength struct {
Offset, Length uint64
}
// not a box
type ItemLocationBoxEntry struct {
ItemID uint16
ConstructionMethod uint8 // actually uint4
DataReferenceIndex uint16
BaseOffset uint64 // uint32 or uint64, depending on encoding
ExtentCount uint16
Extents []OffsetLength
}
// box "iloc"
type ItemLocationBox struct {
FullBox
offsetSize, lengthSize, baseOffsetSize, indexSize uint8 // actually uint4
ItemCount uint16
Items []ItemLocationBoxEntry
}
func parseItemLocationBox(outer *box, br *bufReader) (Box, error) {
fb, err := readFullBox(outer, br)
if err != nil {
return nil, err
}
ilb := &ItemLocationBox{
FullBox: fb,
}
buf, err := br.Peek(4)
if err != nil {
return nil, err
}
ilb.offsetSize = buf[0] >> 4
ilb.lengthSize = buf[0] & 15
ilb.baseOffsetSize = buf[1] >> 4
if fb.Version > 0 { // version 1
ilb.indexSize = buf[1] & 15
}
ilb.ItemCount = binary.BigEndian.Uint16(buf[2:4])
br.Discard(4)
for i := 0; br.ok() && i < int(ilb.ItemCount); i++ {
var ent ItemLocationBoxEntry
ent.ItemID, _ = br.readUint16()
if fb.Version > 0 { // version 1
cmeth, _ := br.readUint16()
ent.ConstructionMethod = byte(cmeth & 15)
}
ent.DataReferenceIndex, _ = br.readUint16()
if br.ok() && ilb.baseOffsetSize > 0 {
br.Discard(int(ilb.baseOffsetSize) / 8)
}
ent.ExtentCount, _ = br.readUint16()
for j := 0; br.ok() && j < int(ent.ExtentCount); j++ {
var ol OffsetLength
ol.Offset, _ = br.readUintN(ilb.offsetSize * 8)
ol.Length, _ = br.readUintN(ilb.lengthSize * 8)
if br.err != nil {
return nil, br.err
}
ent.Extents = append(ent.Extents, ol)
}
ilb.Items = append(ilb.Items, ent)
}
if !br.ok() {
return nil, br.err
}
return ilb, nil
}
// a "hdlr" box.
type HandlerBox struct {
FullBox
HandlerType string // always 4 bytes; usually "pict" for iOS Camera images
Name string
}
func parseHandlerBox(gen *box, br *bufReader) (Box, error) {
fb, err := readFullBox(gen, br)
if err != nil {
return nil, err
}
hb := &HandlerBox{
FullBox: fb,
}
buf, err := br.Peek(20)
if err != nil {
return nil, err
}
hb.HandlerType = string(buf[4:8])
br.Discard(20)
hb.Name, _ = br.readString()
return hb, br.err
}
// a "dinf" box
type DataInformationBox struct {
*box
Children []Box
}
func parseDataInformationBox(gen *box, br *bufReader) (Box, error) {
dib := &DataInformationBox{box: gen}
return dib, br.parseAppendBoxes(&dib.Children)
}
// a "dref" box.
type DataReferenceBox struct {
FullBox
EntryCount uint32
Children []Box
}
func parseDataReferenceBox(gen *box, br *bufReader) (Box, error) {
fb, err := readFullBox(gen, br)
if err != nil {
return nil, err
}
drb := &DataReferenceBox{FullBox: fb}
drb.EntryCount, _ = br.readUint32()
return drb, br.parseAppendBoxes(&drb.Children)
}
// "pitm" box
type PrimaryItemBox struct {
FullBox
ItemID uint16
}
func parsePrimaryItemBox(gen *box, br *bufReader) (Box, error) {
fb, err := readFullBox(gen, br)
if err != nil {
return nil, err
}
pib := &PrimaryItemBox{FullBox: fb}
pib.ItemID, _ = br.readUint16()
if !br.ok() {
return nil, br.err
}
return pib, nil
}
// ImageRotation is a HEIF "irot" rotation property.
type ImageRotation struct {
*box
Angle uint8 // 1 means 90 degrees counter-clockwise, 2 means 180 counter-clockwise
}
func parseImageRotation(gen *box, br *bufReader) (Box, error) {
v, err := br.readUint8()
if err != nil {
return nil, err
}
return &ImageRotation{box: gen, Angle: v & 3}, nil
}
================================================
FILE: media/heif/dumpheif/dumpheif.go
================================================
/*
Copyright 2018 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// The dumpheif program dumps the structure and metadata of a HEIF file.
//
// It exists purely for debugging the go4.org/media/heif and
// go4.org/media/heif/bmff packages; it makes no backwards
// compatibility promises.
package main
import (
"bytes"
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strings"
"github.com/rwcarlsen/goexif/exif"
"github.com/rwcarlsen/goexif/tiff"
"go4.org/media/heif"
"go4.org/media/heif/bmff"
)
var (
exifItemID uint16
exifLoc bmff.ItemLocationBoxEntry
)
func main() {
flag.Parse()
if flag.NArg() != 1 {
fmt.Fprintf(os.Stderr, "usage: dumpheif <file>\n")
os.Exit(1)
}
f, err := os.Open(flag.Arg(0))
if err != nil {
log.Fatal(err)
}
defer f.Close()
hf := heif.Open(f)
it, err := hf.PrimaryItem()
if err != nil {
log.Fatalf("PrimaryItem: %v", err)
}
fmt.Printf("primary item: %v\n", it.ID)
width, height, ok := it.SpatialExtents()
if ok {
fmt.Printf("spatial extents: %d x %d\n", width, height)
}
fmt.Printf("properties:\n")
for _, prop := range it.Properties {
fmt.Printf("\t%q: %#v\n", prop.Type(), prop)
}
if len(it.Properties) == 0 {
fmt.Printf("\t(no properties)\n")
}
if ex, err := hf.EXIF(); err == nil {
fmt.Printf("EXIF dump:\n")
ex, err := exif.Decode(bytes.NewReader(ex))
if err != nil {
log.Fatalf("EXIF decode: %v", err)
}
ex.Walk(exifWalkFunc(func(name exif.FieldName, tag *tiff.Tag) error {
fmt.Printf("\t%v = %v\n", name, tag)
return nil
}))
fmt.Printf("\n")
}
fmt.Printf("BMFF boxes:\n")
r := bmff.NewReader(f)
for {
box, err := r.ReadBox()
if err == io.EOF {
break
}
if err != nil {
log.Fatalf("ReadBox: %v", err)
}
dumpBox(box, 0)
}
}
type exifWalkFunc func(exif.FieldName, *tiff.Tag) error
func (f exifWalkFunc) Walk(name exif.FieldName, tag *tiff.Tag) error {
return f(name, tag)
}
func dumpBox(box bmff.Box, depth int) {
indent := strings.Repeat(" ", depth)
fmt.Printf("%sBox: type %q, size %v\n", indent, box.Type(), box.Size())
box2, err := box.Parse()
if err == bmff.ErrUnknownBox {
slurp, err := ioutil.ReadAll(box.Body())
if err != nil {
log.Fatalf("%sreading body: %v", indent, err)
}
if len(slurp) < 5000 {
fmt.Printf("%s- contents: %q\n", indent, slurp)
} else {
fmt.Printf("%s- contents: (... %d bytes, starting with %q ...)\n", indent, len(slurp), slurp[:100])
}
return
}
if err != nil {
slurp, _ := ioutil.ReadAll(box.Body())
log.Fatalf("Parse box type %q: %v; slurp: %q", box.Type(), err, slurp)
}
switch v := box2.(type) {
case *bmff.FileTypeBox, *bmff.HandlerBox, *bmff.PrimaryItemBox:
fmt.Printf("%s- %T: %+v\n", indent, v, v)
case *bmff.MetaBox:
fmt.Printf("%s- %T, %d children:\n", indent, v, len(v.Children))
for _, child := range v.Children {
dumpBox(child, depth+1)
}
case *bmff.ItemInfoBox:
//slurp, _ := ioutil.ReadAll(box.Body())
//fmt.Printf("%s- %T raw: %q\n", indent, v, slurp)
fmt.Printf("%s- %T, %d children (%d in slice):\n", indent, v, v.Count, len(v.ItemInfos))
for _, child := range v.ItemInfos {
dumpBox(child, depth+1)
}
case *bmff.ItemInfoEntry:
fmt.Printf("%s- %T, %+v\n", indent, v, v)
if v.ItemType == "Exif" {
exifItemID = v.ItemID
}
case *bmff.ItemPropertiesBox:
fmt.Printf("%s- %T\n", indent, v)
if v.PropertyContainer != nil {
dumpBox(v.PropertyContainer, depth+1)
}
for _, child := range v.Associations {
dumpBox(child, depth+1)
}
case *bmff.ItemPropertyAssociation:
fmt.Printf("%s- %T: %d declared entries, %d parsed:\n", indent, v, v.EntryCount, len(v.Entries))
for _, ai := range v.Entries {
fmt.Printf("%s for Item ID %d, %d associations declared, %d parsed:\n", indent, ai.ItemID, ai.AssociationsCount, len(ai.Associations))
for _, ass := range ai.Associations {
fmt.Printf("%s index: %d, essential: %v\n", indent, ass.Index, ass.Essential)
}
}
case *bmff.DataInformationBox:
fmt.Printf("%s- %T\n", indent, v)
for _, child := range v.Children {
dumpBox(child, depth+1)
}
case *bmff.DataReferenceBox:
fmt.Printf("%s- %T\n", indent, v)
for _, child := range v.Children {
dumpBox(child, depth+1)
}
case *bmff.ItemPropertyContainerBox:
fmt.Printf("%s- %T\n", indent, v)
for _, child := range v.Properties {
dumpBox(child, depth+1)
}
case *bmff.ItemLocationBox:
fmt.Printf("%s- %T: %d items declared, %d parsed:\n", indent, v, v.ItemCount, len(v.Items))
for _, lbe := range v.Items {
fmt.Printf("%s %+v\n", indent, lbe)
if exifItemID != 0 && lbe.ItemID == exifItemID {
exifLoc = lbe
}
}
case *bmff.ImageSpatialExtentsProperty:
fmt.Printf("%s- %T dimensions: %d x %d\n", indent, v, v.ImageWidth, v.ImageHeight)
default:
fmt.Printf("%s- gotype: %T\n", indent, box2)
}
}
================================================
FILE: media/heif/heif.go
================================================
/*
Copyright 2018 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package heif reads HEIF containers, as found in Apple HEIC/HEVC images.
// This package does not decode images; it only reads the metadata.
//
// This package is a work in progress and makes no API compatibility
// promises.
package heif
import (
"errors"
"fmt"
"io"
"log"
"go4.org/media/heif/bmff"
)
// File represents a HEIF file.
//
// Methods on File should not be called concurrently.
type File struct {
ra io.ReaderAt
primary *Item
// Populated lazily, by getMeta:
metaErr error
meta *BoxMeta
}
// BoxMeta contains the low-level BMFF metadata boxes.
type BoxMeta struct {
FileType *bmff.FileTypeBox
Handler *bmff.HandlerBox
PrimaryItem *bmff.PrimaryItemBox
ItemInfo *bmff.ItemInfoBox
Properties *bmff.ItemPropertiesBox
ItemLocation *bmff.ItemLocationBox
}
// EXIFItemID returns the item ID of the EXIF part, or 0 if not found.
func (m *BoxMeta) EXIFItemID() uint32 {
if m.ItemInfo == nil {
return 0
}
for _, ife := range m.ItemInfo.ItemInfos {
if ife.ItemType == "Exif" {
return uint32(ife.ItemID)
}
}
return 0
}
// Item represents an item in a HEIF file.
type Item struct {
f *File
ID uint32
Info *bmff.ItemInfoEntry
Location *bmff.ItemLocationBoxEntry // location in file
Properties []bmff.Box
}
// SpatialExtents returns the item's spatial extents property values, if present,
// not correcting from any camera rotation metadata.
func (it *Item) SpatialExtents() (width, height int, ok bool) {
for _, p := range it.Properties {
if p, ok := p.(*bmff.ImageSpatialExtentsProperty); ok {
return int(p.ImageWidth), int(p.ImageHeight), true
}
}
return
}
// Rotations returns the number of 90 degree rotations counter-clockwise that this
// image should be rendered at, in the range [0,3].
func (it *Item) Rotations() int {
for _, p := range it.Properties {
if p, ok := p.(*bmff.ImageRotation); ok {
return int(p.Angle)
}
}
return 0
}
// VisualDimensions returns the item's width and height after correcting
// for any rotations.
func (it *Item) VisualDimensions() (width, height int, ok bool) {
width, height, ok = it.SpatialExtents()
for i := 0; i < it.Rotations(); i++ {
width, height = height, width
}
return
}
// TODO: add HEIF imir (mirroring) accessor, like Image.SpatialExtents.
// Open returns a handle to access a HEIF file.
func Open(f io.ReaderAt) *File {
return &File{ra: f}
}
// ErrNoEXIF is returned by File.EXIF when a file does not contain an EXIF item.
var ErrNoEXIF = errors.New("heif: no EXIF found")
// ErrUnknownItem is returned by File.ItemByID for unknown items.
var ErrUnknownItem = errors.New("heif: unknown item")
// EXIF returns the raw EXIF data from the file.
// The error is ErrNoEXIF if the file did not contain EXIF.
//
// The raw EXIF data can be parsed by the
// github.com/rwcarlsen/goexif/exif package's Decode function.
func (f *File) EXIF() ([]byte, error) {
meta, err := f.getMeta()
if err != nil {
return nil, err
}
exifID := meta.EXIFItemID()
if exifID == 0 {
return nil, ErrNoEXIF
}
it, err := f.ItemByID(exifID)
if err != nil {
return nil, err
}
if it.Location == nil {
return nil, errors.New("heif: file said it contained EXIF, but didn't say where")
}
if n := len(it.Location.Extents); n != 1 {
return nil, fmt.Errorf("heif: expected 1 EXIF section, saw %d", n)
}
offLen := it.Location.Extents[0]
const maxSize = 20 << 10 // 20MB of EXIF seems excessive; cap it for sanity
if offLen.Length > maxSize {
return nil, fmt.Errorf("heif: declared EXIF size %d exceeds threshold of %d bytes", offLen.Length, maxSize)
}
buf := make([]byte, offLen.Length-4)
n, err := f.ra.ReadAt(buf, int64(offLen.Offset)+4) // TODO: why 4? did I miss something?
if err != nil {
log.Printf("Read %d bytes + %v: %q", n, err, buf)
return nil, err
}
return buf, nil
}
func (f *File) setMetaErr(err error) error {
if f.metaErr != nil {
f.metaErr = err
}
return err
}
func (f *File) getMeta() (*BoxMeta, error) {
if f.metaErr != nil {
return nil, f.metaErr
}
if f.meta != nil {
return f.meta, nil
}
const assumedMaxSize = 5 << 40 // arbitrary
sr := io.NewSectionReader(f.ra, 0, assumedMaxSize)
bmr := bmff.NewReader(sr)
meta := &BoxMeta{}
pbox, err := bmr.ReadAndParseBox(bmff.TypeFtyp)
if err != nil {
return nil, f.setMetaErr(err)
}
meta.FileType = pbox.(*bmff.FileTypeBox)
pbox, err = bmr.ReadAndParseBox(bmff.TypeMeta)
if err != nil {
return nil, f.setMetaErr(err)
}
metabox := pbox.(*bmff.MetaBox)
for _, box := range metabox.Children {
boxp, err := box.Parse()
if err == bmff.ErrUnknownBox {
continue
}
if err != nil {
return nil, f.setMetaErr(err)
}
switch v := boxp.(type) {
case *bmff.HandlerBox:
meta.Handler = v
case *bmff.PrimaryItemBox:
meta.PrimaryItem = v
case *bmff.ItemInfoBox:
meta.ItemInfo = v
case *bmff.ItemPropertiesBox:
meta.Properties = v
case *bmff.ItemLocationBox:
meta.ItemLocation = v
}
}
f.meta = meta
return f.meta, nil
}
// PrimaryItem returns the HEIF file's primary item.
func (f *File) PrimaryItem() (*Item, error) {
meta, err := f.getMeta()
if err != nil {
return nil, err
}
if meta.PrimaryItem == nil {
return nil, errors.New("heif: HEIF file lacks primary item box")
}
return f.ItemByID(uint32(meta.PrimaryItem.ItemID))
}
// ItemByID by returns the file's Item of a given ID.
// If the ID is known, the returned error is ErrUnknownItem.
func (f *File) ItemByID(id uint32) (*Item, error) {
meta, err := f.getMeta()
if err != nil {
return nil, err
}
it := &Item{
f: f,
ID: id,
}
if meta.ItemLocation != nil {
for _, ilbe := range meta.ItemLocation.Items {
if uint32(ilbe.ItemID) == id {
shallowCopy := ilbe
it.Location = &shallowCopy
}
}
}
if meta.ItemInfo != nil {
for _, iie := range meta.ItemInfo.ItemInfos {
if uint32(iie.ItemID) == id {
it.Info = iie
}
}
}
if it.Info == nil {
return nil, ErrUnknownItem
}
if meta.Properties != nil {
allProps := meta.Properties.PropertyContainer.Properties
for _, ipa := range meta.Properties.Associations {
// TODO: I've never seen a file with more than
// top-level ItemPropertyAssociation box, but
// apparently they can exist with different
// versions/flags. For now we just merge them
// all together, but that's not really right.
// So for now, just bail once a previous loop
// found anything.
if len(it.Properties) > 0 {
break
}
for _, ipai := range ipa.Entries {
if ipai.ItemID != id {
continue
}
for _, ass := range ipai.Associations {
if ass.Index != 0 && int(ass.Index) <= len(allProps) {
box := allProps[ass.Index-1]
boxp, err := box.Parse()
if err == nil {
box = boxp
}
it.Properties = append(it.Properties, box)
}
}
}
}
}
return it, nil
}
================================================
FILE: media/heif/heif_test.go
================================================
package heif
import (
"bytes"
"fmt"
"os"
"testing"
"github.com/rwcarlsen/goexif/exif"
"github.com/rwcarlsen/goexif/tiff"
)
func TestAll(t *testing.T) {
f, err := os.Open("testdata/park.heic")
if err != nil {
t.Fatal(err)
}
defer f.Close()
h := Open(f)
// meta
_, err = h.getMeta()
if err != nil {
t.Fatalf("getMeta: %v", err)
}
it, err := h.PrimaryItem()
if err != nil {
t.Fatalf("PrimaryItem: %v", err)
}
if want := uint32(49); it.ID != want {
t.Errorf("PrimaryIem ID = %v; want %v", it.ID, want)
}
if it.Location == nil {
t.Errorf("Item.Location is nil")
}
if it.Info == nil {
t.Errorf("Item.Info is nil")
}
if len(it.Properties) == 0 {
t.Errorf("Item.Properties is empty")
}
for _, prop := range it.Properties {
t.Logf(" property: %q, %#v", prop.Type(), prop)
}
if w, h, ok := it.SpatialExtents(); !ok || w == 0 || h == 0 {
t.Errorf("no spatial extents found")
} else {
t.Logf("dimensions: %v x %v", w, h)
}
// exif
exbuf, err := h.EXIF()
if err != nil {
t.Errorf("EXIF: %v", err)
} else {
const magic = "Exif\x00\x00"
if !bytes.HasPrefix(exbuf, []byte(magic)) {
t.Errorf("Exif buffer doesn't start with %q: got %q", magic, exbuf)
}
x, err := exif.Decode(bytes.NewReader(exbuf))
if err != nil {
t.Fatalf("EXIF decode: %v", err)
}
got := map[string]string{}
if err := x.Walk(walkFunc(func(name exif.FieldName, tag *tiff.Tag) error {
got[fmt.Sprint(name)] = fmt.Sprint(tag)
return nil
})); err != nil {
t.Fatalf("EXIF walk: %v", err)
}
if g, w := len(got), 56; g < w {
t.Errorf("saw %v EXIF tags; want at least %v", g, w)
}
if g, w := got["GPSLongitude"], `["122/1","21/1","3776/100"]`; g != w {
t.Errorf("GPSLongitude = %#q; want %#q", g, w)
}
}
}
func TestRotations(t *testing.T) {
f, err := os.Open("testdata/rotate.heic")
if err != nil {
t.Fatal(err)
}
defer f.Close()
h := Open(f)
it, err := h.PrimaryItem()
if err != nil {
t.Fatalf("PrimaryItem: %v", err)
}
if r := it.Rotations(); r != 3 {
t.Errorf("Rotations = %v; want %v", r, 3)
}
sw, sh, ok := it.SpatialExtents()
if !ok {
t.Fatalf("expected spatial extents")
}
vw, vh, ok := it.VisualDimensions()
if !ok {
t.Fatalf("expected visual dimensions")
}
if vw != sh || vh != sw {
t.Errorf("visual dimensions = %v, %v; want %v, %v", vw, vh, sh, sw)
}
}
type walkFunc func(exif.FieldName, *tiff.Tag) error
func (f walkFunc) Walk(name exif.FieldName, tag *tiff.Tag) error {
return f(name, tag)
}
================================================
FILE: must/must.go
================================================
/*
Copyright 2019 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package must contains helpers that panic on failure.
package must // import "go4.org/must"
import "io"
// Close calls c.Close and panics if it returns an error.
// The panic value is the return value from Close.
func Close(c io.Closer) {
if err := c.Close(); err != nil {
panic(err)
}
}
// Do runs fn and panics if it returns an error.
// The panic value is the return value from fn.
func Do(fn func() error) {
if err := fn(); err != nil {
panic(err)
}
}
================================================
FILE: net/throttle/throttle.go
================================================
/*
Copyright 2012 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package throttle provides a net.Listener that returns
// artificially-delayed connections for testing real-world
// connectivity.
package throttle // import "go4.org/net/throttle"
import (
"fmt"
"net"
"sync"
"time"
)
const unitSize = 1400 // read/write chunk size. ~MTU size.
type Rate struct {
KBps int // or 0, to not rate-limit bandwidth
Latency time.Duration
}
// byteTime returns the time required for n bytes.
func (r Rate) byteTime(n int) time.Duration {
if r.KBps == 0 {
return 0
}
return time.Duration(float64(n)/1024/float64(r.KBps)) * time.Second
}
type Listener struct {
net.Listener
Down Rate // server Writes to Client
Up Rate // server Reads from client
}
func (ln *Listener) Accept() (net.Conn, error) {
c, err := ln.Listener.Accept()
time.Sleep(ln.Up.Latency)
if err != nil {
return nil, err
}
tc := &conn{Conn: c, Down: ln.Down, Up: ln.Up}
tc.start()
return tc, nil
}
type nErr struct {
n int
err error
}
type writeReq struct {
writeAt time.Time
p []byte
resc chan nErr
}
type conn struct {
net.Conn
Down Rate // for reads
Up Rate // for writes
wchan chan writeReq
closeOnce sync.Once
closeErr error
}
func (c *conn) start() {
c.wchan = make(chan writeReq, 1024)
go c.writeLoop()
}
func (c *conn) writeLoop() {
for req := range c.wchan {
time.Sleep(req.writeAt.Sub(time.Now()))
var res nErr
for len(req.p) > 0 && res.err == nil {
writep := req.p
if len(writep) > unitSize {
writep = writep[:unitSize]
}
n, err := c.Conn.Write(writep)
time.Sleep(c.Up.byteTime(len(writep)))
res.n += n
res.err = err
req.p = req.p[n:]
}
req.resc <- res
}
}
func (c *conn) Close() error {
c.closeOnce.Do(func() {
err := c.Conn.Close()
close(c.wchan)
c.closeErr = err
})
return c.closeErr
}
func (c *conn) Write(p []byte) (n int, err error) {
defer func() {
if e := recover(); e != nil {
n = 0
err = fmt.Errorf("%v", err)
return
}
}()
resc := make(chan nErr, 1)
c.wchan <- writeReq{time.Now().Add(c.Up.Latency), p, resc}
res := <-resc
return res.n, res.err
}
func (c *conn) Read(p []byte) (n int, err error) {
const max = 1024
if len(p) > max {
p = p[:max]
}
n, err = c.Conn.Read(p)
time.Sleep(c.Down.byteTime(n))
return
}
================================================
FILE: oauthutil/oauth.go
================================================
/*
Copyright 2015 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package oauthutil contains OAuth 2 related utilities.
package oauthutil // import "go4.org/oauthutil"
import (
"encoding/json"
"errors"
"fmt"
"time"
"go4.org/wkfs"
"golang.org/x/oauth2"
)
// TitleBarRedirectURL is the OAuth2 redirect URL to use when the authorization
// code should be returned in the title bar of the browser, with the page text
// prompting the user to copy the code and paste it in the application.
const TitleBarRedirectURL = "urn:ietf:wg:oauth:2.0:oob"
// ErrNoAuthCode is returned when Token() has not found any valid cached token
// and TokenSource does not have an AuthCode for getting a new token.
var ErrNoAuthCode = errors.New("oauthutil: unspecified TokenSource.AuthCode")
// TokenSource is an implementation of oauth2.TokenSource. It uses CacheFile to store and
// reuse the the acquired token, and AuthCode to provide the authorization code that will be
// exchanged for a token otherwise.
type TokenSource struct {
Config *oauth2.Config
// CacheFile is where the token will be stored JSON-encoded. Any call to Token
// first tries to read a valid token from CacheFile.
CacheFile string
// AuthCode provides the authorization code that Token will exchange for a token.
// It usually is a way to prompt the user for the code. If CacheFile does not provide
// a token and AuthCode is nil, Token returns ErrNoAuthCode.
AuthCode func() string
}
var errExpiredToken = errors.New("expired token")
// cachedToken returns the token saved in cacheFile. It specifically returns
// errTokenExpired if the token is expired.
func cachedToken(cacheFile string) (*oauth2.Token, error) {
tok := new(oauth2.Token)
tokenData, err := wkfs.ReadFile(cacheFile)
if err != nil {
return nil, err
}
if err = json.Unmarshal(tokenData, tok); err != nil {
return nil, err
}
if !tok.Valid() {
if tok != nil && time.Now().After(tok.Expiry) {
return nil, errExpiredToken
}
return nil, errors.New("invalid token")
}
return tok, nil
}
// Token first tries to find a valid token in CacheFile, and otherwise uses
// Config and AuthCode to fetch a new token. This new token is saved in CacheFile
// (if not blank). If CacheFile did not provide a token and AuthCode is nil,
// ErrNoAuthCode is returned.
func (src TokenSource) Token() (*oauth2.Token, error) {
var tok *oauth2.Token
var err error
if src.CacheFile != "" {
tok, err = cachedToken(src.CacheFile)
if err == nil {
return tok, nil
}
if err != errExpiredToken {
fmt.Printf("Error getting token from %s: %v\n", src.CacheFile, err)
}
}
if src.AuthCode == nil {
return nil, ErrNoAuthCode
}
tok, err = src.Config.Exchange(oauth2.NoContext, src.AuthCode())
if err != nil {
return nil, fmt.Errorf("could not exchange auth code for a token: %v", err)
}
if src.CacheFile == "" {
return tok, nil
}
tokenData, err := json.Marshal(&tok)
if err != nil {
return nil, fmt.Errorf("could not encode token as json: %v", err)
}
if err := wkfs.WriteFile(src.CacheFile, tokenData, 0600); err != nil {
return nil, fmt.Errorf("could not cache token in %v: %v", src.CacheFile, err)
}
return tok, nil
}
// NewRefreshTokenSource returns a token source that obtains its initial token
// based on the provided config and the refresh token.
func NewRefreshTokenSource(config *oauth2.Config, refreshToken string) oauth2.TokenSource {
var noInitialToken *oauth2.Token = nil
return oauth2.ReuseTokenSource(noInitialToken, config.TokenSource(
oauth2.NoContext, // TODO: maybe accept a context later.
&oauth2.Token{RefreshToken: refreshToken},
))
}
================================================
FILE: osutil/osutil.go
================================================
/*
Copyright 2015 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package osutil contains os level functions.
package osutil // import "go4.org/osutil"
import "os" // capture executable on package init to work around various os issues if
// Executable returns [os.Executable]. This function predates the Go standard
// library's os.Executable and is retained here for compatibility.
//
// Deprecated: use os.Executable directly instead.
func Executable() (string, error) {
return os.Executable()
}
================================================
FILE: readerutil/bufreaderat.go
================================================
/*
Copyright 2018 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import "io"
// NewBufferingReaderAt returns an io.ReaderAt that reads from r as
// necessary and keeps a copy of all data read in memory.
func NewBufferingReaderAt(r io.Reader) io.ReaderAt {
return &bufReaderAt{r: r}
}
type bufReaderAt struct {
r io.Reader
buf []byte
}
func (br *bufReaderAt) ReadAt(p []byte, off int64) (n int, err error) {
endOff := off + int64(len(p))
need := endOff - int64(len(br.buf))
if need > 0 {
buf := make([]byte, need)
var rn int
rn, err = io.ReadFull(br.r, buf)
br.buf = append(br.buf, buf[:rn]...)
}
if int64(len(br.buf)) >= off {
n = copy(p, br.buf[off:])
}
if n == len(p) {
err = nil
}
return
}
================================================
FILE: readerutil/bufreaderat_test.go
================================================
/*
Copyright 2018 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import "testing"
type trackingReader struct {
off int
reads int
readBytes int
}
func (t *trackingReader) Read(p []byte) (n int, err error) {
t.reads++
t.readBytes += len(p)
for len(p) > 0 {
p[0] = '0' + byte(t.off%10)
t.off++
p = p[1:]
n++
}
return
}
func TestBufferingReaderAt(t *testing.T) {
tr := new(trackingReader)
ra := NewBufferingReaderAt(tr)
for i, tt := range []struct {
off int64
want string
wantReads int
wantReadBytes int
}{
{off: 0, want: "0123456789", wantReads: 1, wantReadBytes: 10},
{off: 5, want: "56789", wantReads: 1, wantReadBytes: 10}, // already buffered
{off: 6, want: "67890", wantReads: 2, wantReadBytes: 11}, // need 1 more byte
{off: 0, want: "0123456789", wantReads: 2, wantReadBytes: 11}, // already buffered
} {
got := make([]byte, len(tt.want))
n, err := ra.ReadAt(got, tt.off)
if err != nil || n != len(tt.want) {
t.Errorf("step %d: ReadAt = %v, %v; want %v, %v", i, n, err, len(tt.want), nil)
continue
}
if string(got) != tt.want {
t.Errorf("step %d: ReadAt read %q; want %q", i, got, tt.want)
}
if tr.reads != tt.wantReads {
t.Errorf("step %d: num reads = %d; want %d", i, tr.reads, tt.wantReads)
}
if tr.readBytes != tt.wantReadBytes {
t.Errorf("step %d: read bytes = %d; want %d", i, tr.reads, tt.wantReads)
}
}
}
================================================
FILE: readerutil/countingreader.go
================================================
/*
Copyright 2011 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import "io"
// CountingReader wraps a Reader, incrementing N by the number of
// bytes read. No locking is performed.
type CountingReader struct {
Reader io.Reader
N *int64
}
func (cr CountingReader) Read(p []byte) (n int, err error) {
n, err = cr.Reader.Read(p)
*cr.N += int64(n)
return
}
================================================
FILE: readerutil/fakeseeker.go
================================================
/*
Copyright 2014 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"errors"
"fmt"
"io"
"os"
)
// fakeSeeker can seek to the ends but any read not at the current
// position will fail.
type fakeSeeker struct {
r io.Reader
size int64
fakePos int64
realPos int64
}
// NewFakeSeeker returns a ReadSeeker that can pretend to Seek (based
// on the provided total size of the reader's content), but any reads
// will fail if the fake seek position doesn't match reality.
func NewFakeSeeker(r io.Reader, size int64) io.ReadSeeker {
return &fakeSeeker{r: r, size: size}
}
func (fs *fakeSeeker) Seek(offset int64, whence int) (int64, error) {
var newo int64
switch whence {
default:
return 0, errors.New("invalid whence")
case os.SEEK_SET:
newo = offset
case os.SEEK_CUR:
newo = fs.fakePos + offset
case os.SEEK_END:
newo = fs.size + offset
}
if newo < 0 {
return 0, errors.New("negative seek")
}
fs.fakePos = newo
return newo, nil
}
func (fs *fakeSeeker) Read(p []byte) (n int, err error) {
if fs.fakePos != fs.realPos {
return 0, fmt.Errorf("attempt to read from fake seek offset %d; real offset is %d", fs.fakePos, fs.realPos)
}
n, err = fs.r.Read(p)
fs.fakePos += int64(n)
fs.realPos += int64(n)
return
}
================================================
FILE: readerutil/fakeseeker_test.go
================================================
/*
Copyright 2014 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"os"
"strings"
"testing"
)
func TestFakeSeeker(t *testing.T) {
rs := NewFakeSeeker(strings.NewReader("foobar"), 6)
if pos, err := rs.Seek(0, os.SEEK_END); err != nil || pos != 6 {
t.Fatalf("SEEK_END = %d, %v; want 6, nil", pos, err)
}
if pos, err := rs.Seek(0, os.SEEK_CUR); err != nil || pos != 6 {
t.Fatalf("SEEK_CUR = %d, %v; want 6, nil", pos, err)
}
if pos, err := rs.Seek(0, os.SEEK_SET); err != nil || pos != 0 {
t.Fatalf("SEEK_SET = %d, %v; want 0, nil", pos, err)
}
buf := make([]byte, 3)
if n, err := rs.Read(buf); n != 3 || err != nil || string(buf) != "foo" {
t.Fatalf("First read = %d, %v (buf = %q); want foo", n, err, buf)
}
if pos, err := rs.Seek(0, os.SEEK_CUR); err != nil || pos != 3 {
t.Fatalf("Seek cur pos after first read = %d, %v; want 3, nil", pos, err)
}
if n, err := rs.Read(buf); n != 3 || err != nil || string(buf) != "bar" {
t.Fatalf("Second read = %d, %v (buf = %q); want foo", n, err, buf)
}
if pos, err := rs.Seek(1, os.SEEK_SET); err != nil || pos != 1 {
t.Fatalf("SEEK_SET = %d, %v; want 1, nil", pos, err)
}
const msg = "attempt to read from fake seek offset"
if _, err := rs.Read(buf); err == nil || !strings.Contains(err.Error(), msg) {
t.Fatalf("bogus Read after seek = %v; want something containing %q", err, msg)
}
}
================================================
FILE: readerutil/multireaderat.go
================================================
/*
Copyright 2016 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"io"
"sort"
)
// NewMultiReaderAt is like io.MultiReader but produces a ReaderAt
// (and Size), instead of just a reader.
func NewMultiReaderAt(parts ...SizeReaderAt) SizeReaderAt {
m := &multiRA{
parts: make([]offsetAndSource, 0, len(parts)),
}
var off int64
for _, p := range parts {
m.parts = append(m.parts, offsetAndSource{off, p})
off += p.Size()
}
m.size = off
return m
}
type offsetAndSource struct {
off int64
SizeReaderAt
}
type multiRA struct {
parts []offsetAndSource
size int64
}
func (m *multiRA) Size() int64 { return m.size }
func (m *multiRA) ReadAt(p []byte, off int64) (n int, err error) {
wantN := len(p)
// Skip past the requested offset.
skipParts := sort.Search(len(m.parts), func(i int) bool {
// This function returns whether parts[i] will
// contribute any bytes to our output.
part := m.parts[i]
return part.off+part.Size() > off
})
parts := m.parts[skipParts:]
// How far to skip in the first part.
needSkip := off
if len(parts) > 0 {
needSkip -= parts[0].off
}
for len(parts) > 0 && len(p) > 0 {
readP := p
partSize := parts[0].Size()
if int64(len(readP)) > partSize-needSkip {
readP = readP[:partSize-needSkip]
}
pn, err0 := parts[0].ReadAt(readP, needSkip)
if err0 != nil {
return n, err0
}
n += pn
p = p[pn:]
if int64(pn)+needSkip == partSize {
parts = parts[1:]
}
needSkip = 0
}
if n != wantN {
err = io.ErrUnexpectedEOF
}
return
}
// ZeroSizeReaderAt returns a SizeReaderAt that's size bytes
// of all zeros.
func ZeroSizeReaderAt(size int64) SizeReaderAt {
return allZeros{n: size}
}
type allZeros struct{ n int64 }
func (a allZeros) ReadAt(p []byte, off int64) (n int, err error) {
for i := range p {
p[i] = 0
}
return len(p), nil
}
func (a allZeros) Size() int64 { return a.n }
================================================
FILE: readerutil/multireaderat_test.go
================================================
/*
Copyright 2016 The Go4 Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"io"
"io/ioutil"
"strings"
"testing"
)
func TestMultiReaderAt(t *testing.T) {
sra := NewMultiReaderAt(
io.NewSectionReader(strings.NewReader("xaaax"), 1, 3),
io.NewSectionReader(strings.NewReader("xxbbbbxx"), 2, 3),
io.NewSectionReader(strings.NewReader("cccx"), 0, 3),
)
if sra.Size() != 9 {
t.Fatalf("Size = %d; want 9", sra.Size())
}
const full = "aaabbbccc"
for start := 0; start < len(full); start++ {
for end := start; end < len(full); end++ {
want := full[start:end]
got, err := ioutil.ReadAll(io.NewSectionReader(sra, int64(start), int64(end-start)))
if err != nil {
t.Fatal(err)
}
if string(got) != want {
t.Errorf("for start=%d, end=%d: ReadAll = %q; want %q", start, end, got, want)
}
}
}
}
================================================
FILE: readerutil/readersize.go
================================================
/*
Copyright 2012 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package readerutil provides and operates on io.Readers.
package readerutil // import "go4.org/readerutil"
import (
"bytes"
"io"
"os"
"strings"
)
// Size tries to determine the length of r. If r is an io.Seeker, Size may seek
// to guess the length.
func Size(r io.Reader) (size int64, ok bool) {
switch rt := r.(type) {
case *bytes.Buffer:
return int64(rt.Len()), true
case *bytes.Reader:
return int64(rt.Len()), true
case *strings.Reader:
return int64(rt.Len()), true
case io.Seeker:
pos, err := rt.Seek(0, os.SEEK_CUR)
if err != nil {
return
}
end, err := rt.Seek(0, os.SEEK_END)
if err != nil {
return
}
size = end - pos
pos1, err := rt.Seek(pos, os.SEEK_SET)
if err != nil || pos1 != pos {
msg := "failed to restore seek position"
if err != nil {
msg += ": " + err.Error()
}
panic(msg)
}
return size, true
}
return 0, false
}
================================================
FILE: readerutil/readersize_test.go
================================================
/*
Copyright 2012 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"bytes"
"io"
"io/ioutil"
"os"
"testing"
)
const text = "HelloWorld"
type testSrc struct {
name string
src io.Reader
want int64
}
func (tsrc *testSrc) run(t *testing.T) {
n, ok := Size(tsrc.src)
if !ok {
t.Fatalf("failed to read size for %q", tsrc.name)
}
if n != tsrc.want {
t.Fatalf("wanted %v, got %v", tsrc.want, n)
}
}
func TestBytesBuffer(t *testing.T) {
buf := bytes.NewBuffer([]byte(text))
tsrc := &testSrc{"buffer", buf, int64(len(text))}
tsrc.run(t)
}
func TestSeeker(t *testing.T) {
f, err := ioutil.TempFile("", "camliTestReaderSize")
if err != nil {
t.Fatal(err)
}
defer os.Remove(f.Name())
defer f.Close()
size, err := f.Write([]byte(text))
if err != nil {
t.Fatal(err)
}
pos, err := f.Seek(5, 0)
if err != nil {
t.Fatal(err)
}
tsrc := &testSrc{"seeker", f, int64(size) - pos}
tsrc.run(t)
}
================================================
FILE: readerutil/readerutil.go
================================================
/*
Copyright 2016 The go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package readerutil contains io.Reader types.
package readerutil // import "go4.org/readerutil"
import (
"expvar"
"io"
)
// A SizeReaderAt is a ReaderAt with a Size method.
//
// An io.SectionReader implements SizeReaderAt.
type SizeReaderAt interface {
Size() int64
io.ReaderAt
}
// A ReadSeekCloser can Read, Seek, and Close.
type ReadSeekCloser interface {
io.Reader
io.Seeker
io.Closer
}
type ReaderAtCloser interface {
io.ReaderAt
io.Closer
}
// TODO(wathiede): make sure all the stat readers work with code that
// type asserts ReadFrom/WriteTo.
type varStatReader struct {
*expvar.Int
r io.Reader
}
// NewStatsReader returns an io.Reader that will have the number of bytes
// read from r added to v.
func NewStatsReader(v *expvar.Int, r io.Reader) io.Reader {
return &varStatReader{v, r}
}
func (v *varStatReader) Read(p []byte) (int, error) {
n, err := v.r.Read(p)
v.Int.Add(int64(n))
return n, err
}
type varStatReadSeeker struct {
*expvar.Int
rs io.ReadSeeker
}
// NewStatsReadSeeker returns an io.ReadSeeker that will have the number of bytes
// read from rs added to v.
func NewStatsReadSeeker(v *expvar.Int, rs io.ReadSeeker) io.ReadSeeker {
return &varStatReadSeeker{v, rs}
}
func (v *varStatReadSeeker) Read(p []byte) (int, error) {
n, err := v.rs.Read(p)
v.Int.Add(int64(n))
return n, err
}
func (v *varStatReadSeeker) Seek(offset int64, whence int) (int64, error) {
return v.rs.Seek(offset, whence)
}
================================================
FILE: readerutil/readerutil_test.go
================================================
/*
Copyright 2016 The Go4 Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package readerutil
import (
"expvar"
"fmt"
"io"
"io/ioutil"
"strings"
)
func ExampleNewStatsReader() {
var (
// r is the io.Reader we'd like to count read from.
r = strings.NewReader("Hello world")
v = expvar.NewInt("read-bytes")
sw = NewStatsReader(v, r)
)
// Read from the wrapped io.Reader, StatReader will count the bytes.
io.Copy(ioutil.Discard, sw)
fmt.Printf("Read %s bytes\n", v.String())
// Output: Read 11 bytes
}
================================================
FILE: readerutil/singlereader/opener.go
================================================
/*
Copyright 2013 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// package singlereader provides Open and Close operations, reusing existing
// file descriptors when possible.
package singlereader // import "go4.org/readerutil/singlereader"
import (
"sync"
"go4.org/readerutil"
"go4.org/syncutil/singleflight"
"go4.org/wkfs"
)
var (
openerGroup singleflight.Group
openFileMu sync.Mutex // guards openFiles
openFiles = make(map[string]*openFile)
)
type openFile struct {
wkfs.File
path string // map key of openFiles
refCount int
}
type openFileHandle struct {
closed bool
*openFile
}
func (f *openFileHandle) Close() error {
openFileMu.Lock()
if f.closed {
openFileMu.Unlock()
return nil
}
f.closed = true
f.refCount--
if f.refCount < 0 {
panic("unexpected negative refcount")
}
zero := f.refCount == 0
if zero {
delete(openFiles, f.path)
}
openFileMu.Unlock()
if !zero {
return nil
}
return f.openFile.File.Close()
}
// Open opens the given file path for reading, reusing existing file descriptors
// when possible.
func Open(path string) (readerutil.ReaderAtCloser, error) {
openFileMu.Lock()
of := openFiles[path]
if of != nil {
of.refCount++
openFileMu.Unlock()
return &openFileHandle{false, of}, nil
}
openFileMu.Unlock() // release the lock while we call os.Open
winner := false // this goroutine made it into Do's func
// Returns an *openFile
resi, err := openerGroup.Do(path, func() (interface{}, error) {
winner = true
f, err := wkfs.Open(path)
if err != nil {
return nil, err
}
of := &openFile{
File: f,
path: path,
refCount: 1,
}
openFileMu.Lock()
openFiles[path] = of
openFileMu.Unlock()
return of, nil
})
if err != nil {
return nil, err
}
of = resi.(*openFile)
// If our os.Open was dup-suppressed, we have to increment our
// reference count.
if !winner {
openFileMu.Lock()
if of.refCount == 0 {
// Winner already closed it. Try again (rare).
openFileMu.Unlock()
return Open(path)
}
of.refCount++
openFileMu.Unlock()
}
return &openFileHandle{false, of}, nil
}
================================================
FILE: readerutil/singlereader/opener_test.go
================================================
/*
Copyright 2013 The Go4 Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package singlereader
import (
"bytes"
"fmt"
"io/ioutil"
"os"
"runtime"
"testing"
)
func TestOpenSingle(t *testing.T) {
if testing.Short() {
t.Skip("skipping in short mode")
}
defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(4))
f, err := ioutil.TempFile("", "foo")
if err != nil {
t.Fatal(err)
}
defer os.Remove(f.Name())
contents := []byte("Some file contents")
if _, err := f.Write(contents); err != nil {
t.Fatal(err)
}
f.Close()
const j = 4
errc := make(chan error, j)
for i := 1; i < j; i++ {
go func() {
buf := make([]byte, len(contents))
for i := 0; i < 400; i++ {
rac, err := Open(f.Name())
if err != nil {
errc <- err
return
}
n, err := rac.ReadAt(buf, 0)
if err != nil {
errc <- err
return
}
if n != len(contents) || !bytes.Equal(buf, contents) {
errc <- fmt.Errorf("read %d, %q; want %d, %q", n, buf, len(contents), contents)
return
}
if err := rac.Close(); err != nil {
errc <- err
return
}
}
errc <- nil
}()
}
for i := 1; i < j; i++ {
if err := <-errc; err != nil {
t.Error(err)
}
}
}
================================================
FILE: reflectutil/swapper.go
================================================
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package reflectutil contains a legacy Swapper function
// that has since moved to the Go standard library.
package reflectutil
import "reflect"
// Swapper returns a function which swaps the elements in slice.
// Swapper panics if the provided interface is not a slice.
//
// Its goal is to work safely and efficiently for all versions and
// variants of Go: pre-Go1.5, Go1.5+, safe, unsafe, App Engine,
// GopherJS, etc.
//
// Deprecated: this moved to the Go standard library. Use
// reflect.Swapper in Go 1.8+ instead.
func Swapper(slice any) func(i, j int) {
return reflect.Swapper(slice)
}
================================================
FILE: rollsum/rollsum.go
================================================
/*
Copyright 2011 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package rollsum implements rolling checksums similar to apenwarr's bup, which
// is similar to librsync.
//
// The bup project is at https://github.com/apenwarr/bup and its splitting in
// particular is at https://github.com/apenwarr/bup/blob/master/lib/bup/bupsplit.c
package rollsum // import "go4.org/rollsum"
import (
"math/bits"
)
const windowSize = 64 // Roll assumes windowSize is a power of 2
const charOffset = 31
const blobBits = 13
const blobSize = 1 << blobBits // 8k
type RollSum struct {
s1, s2 uint32
window [windowSize]uint8
wofs int
}
func New() *RollSum {
return &RollSum{
s1: windowSize * charOffset,
s2: windowSize * (windowSize - 1) * charOffset,
}
}
func (rs *RollSum) add(drop, add uint32) {
s1 := rs.s1 + add - drop
rs.s1 = s1
rs.s2 += s1 - uint32(windowSize)*(drop+charOffset)
}
// Roll adds ch to the rolling sum.
func (rs *RollSum) Roll(ch byte) {
wp := &rs.window[rs.wofs]
rs.add(uint32(*wp), uint32(ch))
*wp = ch
rs.wofs = (rs.wofs + 1) & (windowSize - 1)
}
// OnSplit reports whether at least 13 consecutive trailing bits of
// the current checksum are set the same way.
func (rs *RollSum) OnSplit() bool {
return (rs.s2 & (blobSize - 1)) == ((^0) & (blobSize - 1))
}
// OnSplitWithBits reports whether at least n consecutive trailing bits
// of the current checksum are set the same way.
func (rs *RollSum) OnSplitWithBits(n uint32) bool {
mask := (uint32(1) << n) - 1
return rs.s2&mask == (^uint32(0))&mask
}
func (rs *RollSum) Bits() int {
rsum := rs.Digest() >> (blobBits + 1)
return blobBits + bits.TrailingZeros32(^rsum)
}
func (rs *RollSum) Digest() uint32 {
return (rs.s1 << 16) | (rs.s2 & 0xffff)
}
================================================
FILE: rollsum/rollsum_test.go
================================================
/*
Copyright 2011 The Perkeep Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package rollsum
import (
"math/rand"
"testing"
)
func TestSum(t *testing.T) {
var buf [100000]uint8
rnd := rand.New(rand.NewSource(4))
for i := range buf {
buf[i] = uint8(rnd.Intn(256))
}
roll := func(offset, len int) *RollSum {
rs := New()
for count := offset; count < len; count++ {
rs.Roll(buf[count])
}
return rs
}
sum := func(offset, len int) uint32 {
rs := roll(offset, len)
return rs.Digest()
}
sum1a := sum(0, len(buf))
sum1b := sum(1, len(buf))
sum2a := sum(len(buf)-windowSize*5/2, len(buf)-windowSize)
sum2b := sum(0, len(buf)-windowSize)
sum3a := sum(0, windowSize+3)
sum3b := sum(3, windowSize+3)
if sum1a != sum1b {
t.Errorf("sum1a=%d sum1b=%d", sum1a, sum1b)
}
if sum2a != sum2b {
t.Errorf("sum2a=%d sum2b=%d", sum2a, sum2b)
}
if sum3a != sum3b {
t.Errorf("sum3a=%d sum3b=%d", sum3a, sum3b)
}
end := 500
rs := roll(0, windowSize)
for i := 0; i < end; i++ {
sumRoll := rs.Digest()
newRoll := roll(i, i+windowSize).Digest()
if sumRoll != newRoll {
t.Errorf("Error: i=%d, buf[i]=%d, sumRoll=%d, newRoll=%d\n", i, buf[i], sumRoll, newRoll)
}
rs.Roll(buf[i+windowSize])
}
}
func BenchmarkRollsum(b *testing.B) {
const bufSize = 5 << 20
buf := make([]byte, bufSize)
for i := range buf {
buf[i] = byte(rand.Int63())
}
b.ResetTimer()
rs := New()
splits := 0
for i := 0; i < b.N; i++ {
splits = 0
for _, b := range buf {
rs.Roll(b)
if rs.OnSplit() {
_ = rs.Bits()
splits++
}
}
}
b.SetBytes(bufSize)
b.Logf("num splits = %d; every %d bytes", splits, int(float64(bufSize)/float64(splits)))
}
================================================
FILE: sort/example_interface_test.go
================================================
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"go4.org/sort"
)
type Person struct {
Name string
Age int
}
func (p Person) String() string {
return fmt.Sprintf("%s: %d", p.Name, p.Age)
}
// ByAge implements sort.Interface for []Person based on
// the Age field.
type ByAge []Person
func (a ByAge) Len() int { return len(a) }
func (a ByAge) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a ByAge) Less(i, j int) bool { return a[i].Age < a[j].Age }
func ExampleSort() {
people := []Person{
{"Bob", 31},
{"John", 42},
{"Michael", 17},
{"Jenny", 26},
}
fmt.Println(people)
sort.Sort(ByAge(people))
fmt.Println(people)
// Output:
// [Bob: 31 John: 42 Michael: 17 Jenny: 26]
// [Michael: 17 Jenny: 26 Bob: 31 John: 42]
}
================================================
FILE: sort/example_keys_test.go
================================================
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"sort"
)
// A couple of type definitions to make the units clear.
type earthMass float64
type au float64
// A Planet defines the properties of a solar system object.
type Planet struct {
name string
mass earthMass
distance au
}
// By is the type of a "less" function that defines the ordering of its Planet arguments.
type By func(p1, p2 *Planet) bool
// Sort is a method on the function type, By, that sorts the argument slice according to the function.
func (by By) Sort(planets []Planet) {
ps := &planetSorter{
planets: planets,
by: by, // The Sort method's receiver is the function (closure) that defines the sort order.
}
sort.Sort(ps)
}
// planetSorter joins a By function and a slice of Planets to be sorted.
type planetSorter struct {
planets []Planet
by func(p1, p2 *Planet) bool // Closure used in the Less method.
}
// Len is part of sort.Interface.
func (s *planetSorter) Len() int {
return len(s.planets)
}
// Swap is part of sort.Interface.
func (s *planetSorter) Swap(i, j int) {
s.planets[i], s.planets[j] = s.planets[j], s.planets[i]
}
// Less is part of sort.Interface. It is implemented by calling the "by" closure in the sorter.
func (s *planetSorter) Less(i, j int) bool {
return s.by(&s.planets[i], &s.planets[j])
}
var planets = []Planet{
{"Mercury", 0.055, 0.4},
{"Venus", 0.815, 0.7},
{"Earth", 1.0, 1.0},
{"Mars", 0.107, 1.5},
}
// ExampleSortKeys demonstrates a technique for sorting a struct type using programmable sort criteria.
func Example_sortKeys() {
// Closures that order the Planet structure.
name := func(p1, p2 *Planet) bool {
return p1.name < p2.name
}
mass := func(p1, p2 *Planet) bool {
return p1.mass < p2.mass
}
distance := func(p1, p2 *Planet) bool {
return p1.distance < p2.distance
}
decreasingDistance := func(p1, p2 *Planet) bool {
return !distance(p1, p2)
}
// Sort the planets by the various criteria.
By(name).Sort(planets)
fmt.Println("By name:", planets)
By(mass).Sort(planets)
fmt.Println("By mass:", planets)
By(distance).Sort(planets)
fmt.Println("By distance:", planets)
By(decreasingDistance).Sort(planets)
fmt.Println("By decreasing distance:", planets)
// Output: By name: [{Earth 1 1} {Mars 0.107 1.5} {Mercury 0.055 0.4} {Venus 0.815 0.7}]
// By mass: [{Mercury 0.055 0.4} {Mars 0.107 1.5} {Venus 0.815 0.7} {Earth 1 1}]
// By distance: [{Mercury 0.055 0.4} {Venus 0.815 0.7} {Earth 1 1} {Mars 0.107 1.5}]
// By decreasing distance: [{Mars 0.107 1.5} {Earth 1 1} {Venus 0.815 0.7} {Mercury 0.055 0.4}]
}
================================================
FILE: sort/example_multi_test.go
================================================
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"sort"
)
// A Change is a record of source code changes, recording user, language, and delta size.
type Change struct {
user string
language string
lines int
}
type lessFunc func(p1, p2 *Change) bool
// multiSorter implements the Sort interface, sorting the changes within.
type multiSorter struct {
changes []Change
less []lessFunc
}
// Sort sorts the argument slice according to the less functions passed to OrderedBy.
func (ms *multiSorter) Sort(changes []Change) {
ms.changes = changes
sort.Stable(ms)
}
// OrderedBy returns a Sorter that sorts using the less functions, in order.
// Call its Sort method to sort the data.
func OrderedBy(less ...lessFunc) *multiSorter {
return &multiSorter{
less: less,
}
}
// Len is part of sort.Interface.
func (ms *multiSorter) Len() int {
return len(ms.changes)
}
// Swap is part of sort.Interface.
func (ms *multiSorter) Swap(i, j int) {
ms.changes[i], ms.changes[j] = ms.changes[j], ms.changes[i]
}
// Less is part of sort.Interface. It is implemented by looping along the
// less functions until it finds a comparison that is either Less or
// !Less. Note that it can call the less functions twice per call. We
// could change the functions to return -1, 0, 1 and reduce the
// number of calls for greater efficiency: an exercise for the reader.
func (ms *multiSorter) Less(i, j int) bool {
p, q := &ms.changes[i], &ms.changes[j]
// Try all but the last comparison.
var k int
for k = 0; k < len(ms.less)-1; k++ {
less := ms.less[k]
switch {
case less(p, q):
// p < q, so we have a decision.
return true
case less(q, p):
// p > q, so we have a decision.
return false
}
// p == q; try the next comparison.
}
// All comparisons to here said "equal", so just return whatever
// the final comparison reports.
return ms.less[k](p, q)
}
var changes = []Change{
{"gri", "Go", 100},
{"ken", "C", 150},
{"glenda", "Go", 200},
{"rsc", "Go", 200},
{"r", "Go", 100},
{"ken", "Go", 200},
{"dmr", "C", 100},
{"r", "C", 150},
{"gri", "Smalltalk", 80},
}
// ExampleMultiKeys demonstrates a technique for sorting a struct type using different
// sets of multiple fields in the comparison. We chain together "Less" functions, each of
// which compares a single field.
func Example_sortMultiKeys() {
// Closures that order the Change structure.
user := func(c1, c2 *Change) bool {
return c1.user < c2.user
}
language := func(c1, c2 *Change) bool {
return c1.language < c2.language
}
increasingLines := func(c1, c2 *Change) bool {
return c1.lines < c2.lines
}
decreasingLines := func(c1, c2 *Change) bool {
return c1.lines > c2.lines // Note: > orders downwards.
}
// Simple use: Sort by user.
OrderedBy(user).Sort(changes)
fmt.Println("By user:", changes)
// More examples.
OrderedBy(user, increasingLines).Sort(changes)
fmt.Println("By user,<lines:", changes)
OrderedBy(user, decreasingLines).Sort(changes)
fmt.Println("By user,>lines:", changes)
OrderedBy(language, increasingLines).Sort(changes)
fmt.Println("By language,<lines:", changes)
OrderedBy(language, increasingLines, user).Sort(changes)
fmt.Println("By language,<lines,user:", changes)
// Output:
// By user: [{dmr C 100} {glenda Go 200} {gri Go 100} {gri Smalltalk 80} {ken C 150} {ken Go 200} {r Go 100} {r C 150} {rsc Go 200}]
// By user,<lines: [{dmr C 100} {glenda Go 200} {gri Smalltalk 80} {gri Go 100} {ken C 150} {ken Go 200} {r Go 100} {r C 150} {rsc Go 200}]
// By user,>lines: [{dmr C 100} {glenda Go 200} {gri Go 100} {gri Smalltalk 80} {ken Go 200} {ken C 150} {r C 150} {r Go 100} {rsc Go 200}]
// By language,<lines: [{dmr C 100} {ken C 150} {r C 150} {gri Go 100} {r Go 100} {glenda Go 200} {ken Go 200} {rsc Go 200} {gri Smalltalk 80}]
// By language,<lines,user: [{dmr C 100} {ken C 150} {r C 150} {gri Go 100} {r Go 100} {glenda Go 200} {ken Go 200} {rsc Go 200} {gri Smalltalk 80}]
}
================================================
FILE: sort/example_slice_test.go
================================================
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"go4.org/sort"
)
func Example() {
people := []Person{
{Name: "Bob", Age: 31},
{Name: "John", Age: 42},
{Name: "Michael", Age: 17},
{Name: "Jenny", Age: 26},
}
fmt.Println(people)
sort.Slice(people, func(i, j int) bool { return people[i].Age < people[j].Age })
fmt.Println(people)
// Output:
// [Bob: 31 John: 42 Michael: 17 Jenny: 26]
// [Michael: 17 Jenny: 26 Bob: 31 John: 42]
}
func ExampleSlice() {
people := []Person{
{Name: "Bob", Age: 31},
{Name: "John", Age: 42},
{Name: "Michael", Age: 17},
{Name: "Jenny", Age: 26},
}
fmt.Println(people)
sort.Slice(people, func(i, j int) bool { return people[i].Age < people[j].Age })
fmt.Println(people)
// Output:
// [Bob: 31 John: 42 Michael: 17 Jenny: 26]
// [Michael: 17 Jenny: 26 Bob: 31 John: 42]
}
================================================
FILE: sort/example_test.go
================================================
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"sort"
)
func ExampleInts() {
s := []int{5, 2, 6, 3, 1, 4} // unsorted
sort.Ints(s)
fmt.Println(s)
// Output: [1 2 3 4 5 6]
}
func ExampleReverse() {
s := []int{5, 2, 6, 3, 1, 4} // unsorted
sort.Sort(sort.Reverse(sort.IntSlice(s)))
fmt.Println(s)
// Output: [6 5 4 3 2 1]
}
================================================
FILE: sort/example_wrapper_test.go
================================================
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package sort_test
import (
"fmt"
"sort"
)
type Grams int
func (g Grams) String() string { return fmt.Sprintf("%dg", int(g)) }
type Organ struct {
Name string
Weight Grams
}
type Organs []*Organ
func (s Organs
gitextract_mezihb_5/
├── .gitignore
├── .travis.yml
├── AUTHORS
├── LICENSE
├── README.md
├── bytereplacer/
│ ├── bytereplacer.go
│ └── bytereplacer_test.go
├── cloud/
│ ├── cloudlaunch/
│ │ └── cloudlaunch.go
│ └── google/
│ ├── gceutil/
│ │ └── gceutil.go
│ └── gcsutil/
│ └── storage.go
├── ctxutil/
│ └── ctxutil.go
├── errorutil/
│ └── highlight.go
├── fault/
│ └── fault.go
├── go.mod
├── go.sum
├── go4test/
│ └── cloudlaunch/
│ └── serve_on_cloud.go
├── jsonconfig/
│ ├── eval.go
│ ├── jsonconfig.go
│ ├── jsonconfig_test.go
│ └── testdata/
│ ├── boolenv.json
│ ├── include1.json
│ ├── include1bis.json
│ ├── include2.json
│ ├── listexpand.json
│ ├── loop1.json
│ └── loop2.json
├── legal/
│ ├── legal.go
│ └── legal_test.go
├── lock/
│ ├── .gitignore
│ ├── lock.go
│ ├── lock_plan9.go
│ ├── lock_sigzero.go
│ ├── lock_test.go
│ ├── lock_unix.go
│ └── lock_windows.go
├── media/
│ └── heif/
│ ├── bmff/
│ │ └── bmff.go
│ ├── dumpheif/
│ │ └── dumpheif.go
│ ├── heif.go
│ ├── heif_test.go
│ └── testdata/
│ ├── park.heic
│ └── rotate.heic
├── must/
│ └── must.go
├── net/
│ └── throttle/
│ └── throttle.go
├── oauthutil/
│ └── oauth.go
├── osutil/
│ └── osutil.go
├── readerutil/
│ ├── bufreaderat.go
│ ├── bufreaderat_test.go
│ ├── countingreader.go
│ ├── fakeseeker.go
│ ├── fakeseeker_test.go
│ ├── multireaderat.go
│ ├── multireaderat_test.go
│ ├── readersize.go
│ ├── readersize_test.go
│ ├── readerutil.go
│ ├── readerutil_test.go
│ └── singlereader/
│ ├── opener.go
│ └── opener_test.go
├── reflectutil/
│ └── swapper.go
├── rollsum/
│ ├── rollsum.go
│ └── rollsum_test.go
├── sort/
│ ├── example_interface_test.go
│ ├── example_keys_test.go
│ ├── example_multi_test.go
│ ├── example_slice_test.go
│ ├── example_test.go
│ ├── example_wrapper_test.go
│ ├── export_test.go
│ ├── genzfunc.go
│ ├── search.go
│ ├── search_test.go
│ ├── sort.go
│ ├── sort_test.go
│ └── zfuncversion.go
├── strutil/
│ ├── intern.go
│ ├── strconv.go
│ ├── strutil.go
│ └── strutil_test.go
├── syncutil/
│ ├── gate.go
│ ├── group.go
│ ├── once.go
│ ├── once_test.go
│ ├── sem.go
│ ├── sem_test.go
│ ├── singleflight/
│ │ ├── singleflight.go
│ │ └── singleflight_test.go
│ ├── syncdebug/
│ │ ├── syncdebug.go
│ │ └── syncdebug_test.go
│ └── syncutil.go
├── testing/
│ └── functest/
│ ├── functest.go
│ └── functest_test.go
├── types/
│ ├── types.go
│ └── types_test.go
├── wkfs/
│ ├── gcs/
│ │ ├── gcs.go
│ │ └── gcs_test.go
│ └── wkfs.go
├── writerutil/
│ ├── writerutil.go
│ └── writerutil_test.go
├── xdgdir/
│ ├── example_test.go
│ ├── xdgdir.go
│ └── xdgdir_test.go
└── ziputil/
├── ziputil.go
└── ziputil_test.go
SYMBOL INDEX (775 symbols across 85 files)
FILE: bytereplacer/bytereplacer.go
type Replacer (line 24) | type Replacer struct
method Replace (line 72) | func (r *Replacer) Replace(s []byte) []byte {
type replacer (line 29) | type replacer interface
function New (line 36) | func New(oldnew ...string) *Replacer {
type trieNode (line 76) | type trieNode struct
method add (line 84) | func (t *trieNode) add(key, val []byte, priority int, r *genericReplac...
type genericReplacer (line 185) | type genericReplacer struct
method lookup (line 147) | func (r *genericReplacer) lookup(s []byte, ignoreRoot bool) (val []byt...
method Replace (line 226) | func (r *genericReplacer) Replace(s []byte) []byte {
function makeGenericReplacer (line 194) | func makeGenericReplacer(oldnew []string) *genericReplacer {
type byteReplacer (line 279) | type byteReplacer
method Replace (line 281) | func (r *byteReplacer) Replace(s []byte) []byte {
FILE: bytereplacer/bytereplacer_test.go
function TestReplacer (line 43) | func TestReplacer(t *testing.T) {
function BenchmarkGenericNoMatch (line 310) | func BenchmarkGenericNoMatch(b *testing.B) {
function BenchmarkGenericMatch1 (line 318) | func BenchmarkGenericMatch1(b *testing.B) {
function BenchmarkGenericMatch2 (line 326) | func BenchmarkGenericMatch2(b *testing.B) {
function benchmarkSingleString (line 333) | func benchmarkSingleString(b *testing.B, pattern, text string) {
function BenchmarkSingleMaxSkipping (line 344) | func BenchmarkSingleMaxSkipping(b *testing.B) {
function BenchmarkSingleLongSuffixFail (line 348) | func BenchmarkSingleLongSuffixFail(b *testing.B) {
function BenchmarkSingleMatch (line 352) | func BenchmarkSingleMatch(b *testing.B) {
function benchmarkReplacer (line 356) | func benchmarkReplacer(b *testing.B, r *Replacer, str string) {
function BenchmarkByteByteNoMatch (line 365) | func BenchmarkByteByteNoMatch(b *testing.B) {
function BenchmarkByteByteMatch (line 369) | func BenchmarkByteByteMatch(b *testing.B) {
function BenchmarkByteStringMatch (line 373) | func BenchmarkByteStringMatch(b *testing.B) {
function BenchmarkHTMLEscapeNew (line 377) | func BenchmarkHTMLEscapeNew(b *testing.B) {
function BenchmarkHTMLEscapeOld (line 381) | func BenchmarkHTMLEscapeOld(b *testing.B) {
function oldHTMLEscape (line 391) | func oldHTMLEscape(s []byte) []byte {
function BenchmarkByteByteReplaces (line 401) | func BenchmarkByteByteReplaces(b *testing.B) {
function BenchmarkByteByteMap (line 409) | func BenchmarkByteByteMap(b *testing.B) {
FILE: cloud/cloudlaunch/cloudlaunch.go
function readFile (line 49) | func readFile(v string) string {
constant baseConfig (line 57) | baseConfig = `#cloud-config
type RestartPolicy (line 83) | type RestartPolicy
constant RestartOnUpdates (line 86) | RestartOnUpdates RestartPolicy = iota
constant RestartNever (line 87) | RestartNever
type Config (line 92) | type Config struct
method binaryURL (line 132) | func (c *Config) binaryURL() string {
method instName (line 136) | func (c *Config) instName() string { return c.Name }
method zone (line 137) | func (c *Config) zone() string { return strDefault(c.Zone, "...
method machineType (line 138) | func (c *Config) machineType() string { return strDefault(c.Machine...
method binaryObject (line 139) | func (c *Config) binaryObject() string { return strDefault(c.BinaryO...
method updateStrategy (line 140) | func (c *Config) updateStrategy() string { return strDefault(c.UpdateS...
method projectAPIURL (line 142) | func (c *Config) projectAPIURL() string {
method machineTypeURL (line 145) | func (c *Config) machineTypeURL() string {
method MaybeDeploy (line 160) | func (c *Config) MaybeDeploy() {
method restartLoop (line 197) | func (c *Config) restartLoop() {
type cloudLaunch (line 126) | type cloudLaunch struct
method uploadBinary (line 232) | func (cl *cloudLaunch) uploadBinary() {
method findIP (line 308) | func (cl *cloudLaunch) findIP() string {
method createInstance (line 329) | func (cl *cloudLaunch) createInstance() {
method lookupInstance (line 426) | func (cl *cloudLaunch) lookupInstance() *compute.Instance {
method instanceDisk (line 436) | func (cl *cloudLaunch) instanceDisk() *compute.AttachedDisk {
function strDefault (line 149) | func strDefault(a, b string) string {
function getSelfPath (line 278) | func getSelfPath() string {
function zoneInRegion (line 289) | func zoneInRegion(zone, regionURL string) bool {
FILE: cloud/google/gceutil/gceutil.go
function CoreOSImageURL (line 33) | func CoreOSImageURL(cl *http.Client) (string, error) {
function COSImageURL (line 39) | func COSImageURL(cl *http.Client) (string, error) {
function osImageURL (line 43) | func osImageURL(cl *http.Client, cos bool) (string, error) {
type InstanceGroupAndManager (line 98) | type InstanceGroupAndManager struct
function InstanceGroups (line 108) | func InstanceGroups(svc *compute.Service, proj, zone string) (map[string...
FILE: cloud/google/gcsutil/storage.go
constant gsAccessURL (line 36) | gsAccessURL = "https://storage.googleapis.com"
type Object (line 40) | type Object struct
method valid (line 45) | func (o *Object) valid() error {
method String (line 64) | func (o *Object) String() string {
type SizedObject (line 59) | type SizedObject struct
method String (line 71) | func (so SizedObject) String() string {
function simpleRequest (line 76) | func simpleRequest(method, url_ string) (*http.Request, error) {
function GetPartialObject (line 94) | func GetPartialObject(ctx context.Context, obj Object, offset, length in...
function EnumerateObjects (line 136) | func EnumerateObjects(ctx context.Context, bucket, after string, limit i...
FILE: ctxutil/ctxutil.go
function Client (line 37) | func Client(ctx context.Context) *http.Client {
FILE: errorutil/highlight.go
function HighlightBytePosition (line 32) | func HighlightBytePosition(f io.Reader, pos int64) (line, col int, highl...
FILE: fault/fault.go
type Injector (line 31) | type Injector struct
method ShouldFail (line 47) | func (in *Injector) ShouldFail() bool {
method FailErr (line 53) | func (in *Injector) FailErr(err *error) bool {
function NewInjector (line 39) | func NewInjector(name string) *Injector {
FILE: go4test/cloudlaunch/serve_on_cloud.go
function serveHTTP (line 47) | func serveHTTP(w http.ResponseWriter, r *http.Request) {
function main (line 57) | func main() {
FILE: jsonconfig/eval.go
type stringVector (line 36) | type stringVector struct
method Push (line 40) | func (v *stringVector) Push(s string) {
method Pop (line 44) | func (v *stringVector) Pop() {
method Last (line 48) | func (v *stringVector) Last() string {
type File (line 53) | type File interface
type ConfigParser (line 61) | type ConfigParser struct
method open (line 76) | func (c *ConfigParser) open(filename string) (File, error) {
method ReadFile (line 88) | func (c *ConfigParser) ReadFile(path string) (Obj, error) {
method recursiveReadJSON (line 99) | func (c *ConfigParser) recursiveReadJSON(configPath string) (decodedOb...
method evalValue (line 173) | func (c *ConfigParser) evalValue(v interface{}) (interface{}, error) {
method CheckTypes (line 199) | func (c *ConfigParser) CheckTypes(m map[string]interface{}) error {
method evaluateExpressions (line 205) | func (c *ConfigParser) evaluateExpressions(m map[string]interface{}, s...
method expandEnv (line 236) | func (c *ConfigParser) expandEnv(v []interface{}) (interface{}, error) {
method expandFile (line 284) | func (c *ConfigParser) expandFile(v []interface{}) (exp interface{}, e...
method ConfigFilePath (line 304) | func (c *ConfigParser) ConfigFilePath(configFile string) (path string,...
function RegisterFunc (line 150) | func RegisterFunc(name string, fn func(c *ConfigParser, v []interface{})...
type expanderFunc (line 160) | type expanderFunc
function namedExpander (line 162) | func namedExpander(name string) (fn expanderFunc, ok bool) {
FILE: jsonconfig/jsonconfig.go
type Obj (line 30) | type Obj
method RequiredObject (line 40) | func (jc Obj) RequiredObject(key string) Obj {
method OptionalObject (line 44) | func (jc Obj) OptionalObject(key string) Obj {
method obj (line 48) | func (jc Obj) obj(key string, optional bool) Obj {
method RequiredString (line 66) | func (jc Obj) RequiredString(key string) string {
method OptionalString (line 70) | func (jc Obj) OptionalString(key, def string) string {
method string (line 74) | func (jc Obj) string(key string, def *string) string {
method RequiredStringOrObject (line 92) | func (jc Obj) RequiredStringOrObject(key string) interface{} {
method OptionalStringOrObject (line 96) | func (jc Obj) OptionalStringOrObject(key string) interface{} {
method stringOrObject (line 100) | func (jc Obj) stringOrObject(key string, required bool) interface{} {
method RequiredBool (line 120) | func (jc Obj) RequiredBool(key string) bool {
method OptionalBool (line 124) | func (jc Obj) OptionalBool(key string, def bool) bool {
method bool (line 128) | func (jc Obj) bool(key string, def *bool) bool {
method RequiredInt (line 153) | func (jc Obj) RequiredInt(key string) int {
method OptionalInt (line 157) | func (jc Obj) OptionalInt(key string, def int) int {
method int (line 161) | func (jc Obj) int(key string, def *int) int {
method RequiredInt64 (line 179) | func (jc Obj) RequiredInt64(key string) int64 {
method OptionalInt64 (line 183) | func (jc Obj) OptionalInt64(key string, def int64) int64 {
method int64 (line 187) | func (jc Obj) int64(key string, def *int64) int64 {
method RequiredList (line 205) | func (jc Obj) RequiredList(key string) []string {
method OptionalList (line 209) | func (jc Obj) OptionalList(key string) []string {
method requiredList (line 213) | func (jc Obj) requiredList(key string, required bool) []string {
method noteKnownKey (line 239) | func (jc Obj) noteKnownKey(key string) {
method appendError (line 247) | func (jc Obj) appendError(err error) {
method UnknownKeys (line 257) | func (jc Obj) UnknownKeys() []string {
method Validate (line 279) | func (jc Obj) Validate() error {
function ReadFile (line 35) | func ReadFile(configPath string) (Obj, error) {
FILE: jsonconfig/jsonconfig_test.go
function testIncludes (line 26) | func testIncludes(configFile string, t *testing.T) {
function TestIncludesCWD (line 42) | func TestIncludesCWD(t *testing.T) {
function TestIncludesIncludeDirs (line 46) | func TestIncludesIncludeDirs(t *testing.T) {
function TestIncludeLoop (line 50) | func TestIncludeLoop(t *testing.T) {
function TestBoolEnvs (line 60) | func TestBoolEnvs(t *testing.T) {
function TestListExpansion (line 96) | func TestListExpansion(t *testing.T) {
FILE: legal/legal.go
function RegisterLicense (line 24) | func RegisterLicense(text string) {
function Licenses (line 30) | func Licenses() []string {
FILE: legal/legal_test.go
function TestRegisterLicense (line 23) | func TestRegisterLicense(t *testing.T) {
FILE: lock/lock.go
function Lock (line 47) | func Lock(name string) (io.Closer, error) {
function lockPortable (line 70) | func lockPortable(name string) (io.Closer, error) {
type lockStatus (line 97) | type lockStatus
constant statusInvalid (line 100) | statusInvalid lockStatus = iota
constant statusLocked (line 101) | statusLocked
constant statusUnlocked (line 102) | statusUnlocked
constant statusStale (line 103) | statusStale
type pidLockMeta (line 106) | type pidLockMeta struct
function portableLockStatus (line 110) | func portableLockStatus(path string) lockStatus {
type unlocker (line 145) | type unlocker struct
method Close (line 155) | func (u *unlocker) Close() error {
method close (line 160) | func (u *unlocker) close() {
FILE: lock/lock_plan9.go
function init (line 25) | func init() {
function lockPlan9 (line 29) | func lockPlan9(name string) (io.Closer, error) {
FILE: lock/lock_sigzero.go
function init (line 23) | func init() {
FILE: lock/lock_test.go
function TestLock (line 32) | func TestLock(t *testing.T) {
function TestLockPortable (line 36) | func TestLockPortable(t *testing.T) {
function TestLockInChild (line 40) | func TestLockInChild(t *testing.T) {
function testLock (line 74) | func testLock(t *testing.T, portable bool) {
type childLockCmd (line 150) | type childLockCmd struct
type childProc (line 155) | type childProc struct
method kill (line 160) | func (c *childProc) kill() {
method do (line 164) | func (c *childProc) do(op string) error {
function newChildProc (line 173) | func newChildProc(t *testing.T, path string, portable bool) *childProc {
FILE: lock/lock_unix.go
function init (line 29) | func init() {
function lockFcntl (line 33) | func lockFcntl(name string) (io.Closer, error) {
FILE: lock/lock_windows.go
function init (line 28) | func init() {
type winUnlocker (line 32) | type winUnlocker struct
method Close (line 41) | func (u *winUnlocker) Close() error {
method close (line 46) | func (u *winUnlocker) close() {
function lockWindows (line 54) | func lockWindows(name string) (io.Closer, error) {
function winCreateEphemeral (line 68) | func winCreateEphemeral(name string) (windows.Handle, error) {
FILE: media/heif/bmff/bmff.go
function NewReader (line 39) | func NewReader(r io.Reader) *Reader {
type Reader (line 47) | type Reader struct
method ReadBox (line 167) | func (r *Reader) ReadBox() (Box, error) {
method ReadAndParseBox (line 228) | func (r *Reader) ReadAndParseBox(typ BoxType) (Box, error) {
type BoxType (line 53) | type BoxType
method String (line 61) | func (t BoxType) String() string { return string(t[:]) }
method EqualString (line 63) | func (t BoxType) EqualString(s string) bool {
type parseFunc (line 68) | type parseFunc
type Box (line 71) | type Box interface
type parserFunc (line 95) | type parserFunc
function boxType (line 97) | func boxType(s string) BoxType {
type box (line 121) | type box struct
method Size (line 129) | func (b *box) Size() int64 { return b.size }
method Type (line 130) | func (b *box) Type() BoxType { return b.boxType }
method Body (line 132) | func (b *box) Body() io.Reader {
method Parse (line 139) | func (b *box) Parse() (Box, error) {
type FullBox (line 155) | type FullBox struct
function readFullBox (line 243) | func readFullBox(outer *box, br *bufReader) (fb FullBox, err error) {
type FileTypeBox (line 257) | type FileTypeBox struct
function parseFileTypeBox (line 264) | func parseFileTypeBox(outer *box, br *bufReader) (Box, error) {
type MetaBox (line 288) | type MetaBox struct
function parseMetaBox (line 293) | func parseMetaBox(outer *box, br *bufReader) (Box, error) {
type ItemInfoEntry (line 329) | type ItemInfoEntry struct
function parseItemInfoEntry (line 346) | func parseItemInfoEntry(outer *box, br *bufReader) (Box, error) {
type ItemInfoBox (line 384) | type ItemInfoBox struct
function parseItemInfoBox (line 390) | func parseItemInfoBox(outer *box, br *bufReader) (Box, error) {
type bufReader (line 419) | type bufReader struct
method parseAppendBoxes (line 302) | func (br *bufReader) parseAppendBoxes(dst *[]Box) error {
method ok (line 425) | func (br *bufReader) ok() bool { return br.err == nil }
method anyRemain (line 427) | func (br *bufReader) anyRemain() bool {
method readUintN (line 435) | func (br *bufReader) readUintN(bits uint8) (uint64, error) {
method readUint8 (line 464) | func (br *bufReader) readUint8() (uint8, error) {
method readUint16 (line 476) | func (br *bufReader) readUint16() (uint16, error) {
method readUint32 (line 490) | func (br *bufReader) readUint32() (uint32, error) {
method readString (line 504) | func (br *bufReader) readString() (string, error) {
type ItemPropertyContainerBox (line 523) | type ItemPropertyContainerBox struct
function parseItemPropertyContainerBox (line 528) | func parseItemPropertyContainerBox(outer *box, br *bufReader) (Box, erro...
type ItemPropertiesBox (line 534) | type ItemPropertiesBox struct
function parseItemPropertiesBox (line 540) | func parseItemPropertiesBox(outer *box, br *bufReader) (Box, error) {
type ItemPropertyAssociation (line 581) | type ItemPropertyAssociation struct
type ItemProperty (line 588) | type ItemProperty struct
type ItemPropertyAssociationItem (line 594) | type ItemPropertyAssociationItem struct
function parseItemPropertyAssociation (line 600) | func parseItemPropertyAssociation(outer *box, br *bufReader) (Box, error) {
type ImageSpatialExtentsProperty (line 647) | type ImageSpatialExtentsProperty struct
function parseImageSpatialExtentsProperty (line 653) | func parseImageSpatialExtentsProperty(outer *box, br *bufReader) (Box, e...
type OffsetLength (line 673) | type OffsetLength struct
type ItemLocationBoxEntry (line 678) | type ItemLocationBoxEntry struct
type ItemLocationBox (line 688) | type ItemLocationBox struct
function parseItemLocationBox (line 697) | func parseItemLocationBox(outer *box, br *bufReader) (Box, error) {
type HandlerBox (line 749) | type HandlerBox struct
function parseHandlerBox (line 755) | func parseHandlerBox(gen *box, br *bufReader) (Box, error) {
type DataInformationBox (line 775) | type DataInformationBox struct
function parseDataInformationBox (line 780) | func parseDataInformationBox(gen *box, br *bufReader) (Box, error) {
type DataReferenceBox (line 786) | type DataReferenceBox struct
function parseDataReferenceBox (line 792) | func parseDataReferenceBox(gen *box, br *bufReader) (Box, error) {
type PrimaryItemBox (line 803) | type PrimaryItemBox struct
function parsePrimaryItemBox (line 808) | func parsePrimaryItemBox(gen *box, br *bufReader) (Box, error) {
type ImageRotation (line 822) | type ImageRotation struct
function parseImageRotation (line 827) | func parseImageRotation(gen *box, br *bufReader) (Box, error) {
FILE: media/heif/dumpheif/dumpheif.go
function main (line 46) | func main() {
type exifWalkFunc (line 106) | type exifWalkFunc
method Walk (line 108) | func (f exifWalkFunc) Walk(name exif.FieldName, tag *tiff.Tag) error {
function dumpBox (line 112) | func dumpBox(box bmff.Box, depth int) {
FILE: media/heif/heif.go
type File (line 36) | type File struct
method EXIF (line 128) | func (f *File) EXIF() ([]byte, error) {
method setMetaErr (line 161) | func (f *File) setMetaErr(err error) error {
method getMeta (line 168) | func (f *File) getMeta() (*BoxMeta, error) {
method PrimaryItem (line 220) | func (f *File) PrimaryItem() (*Item, error) {
method ItemByID (line 233) | func (f *File) ItemByID(id uint32) (*Item, error) {
type BoxMeta (line 46) | type BoxMeta struct
method EXIFItemID (line 56) | func (m *BoxMeta) EXIFItemID() uint32 {
type Item (line 69) | type Item struct
method SpatialExtents (line 80) | func (it *Item) SpatialExtents() (width, height int, ok bool) {
method Rotations (line 91) | func (it *Item) Rotations() int {
method VisualDimensions (line 102) | func (it *Item) VisualDimensions() (width, height int, ok bool) {
function Open (line 113) | func Open(f io.ReaderAt) *File {
FILE: media/heif/heif_test.go
function TestAll (line 13) | func TestAll(t *testing.T) {
function TestRotations (line 82) | func TestRotations(t *testing.T) {
type walkFunc (line 109) | type walkFunc
method Walk (line 111) | func (f walkFunc) Walk(name exif.FieldName, tag *tiff.Tag) error {
FILE: must/must.go
function Close (line 24) | func Close(c io.Closer) {
function Do (line 32) | func Do(fn func() error) {
FILE: net/throttle/throttle.go
constant unitSize (line 29) | unitSize = 1400
type Rate (line 31) | type Rate struct
method byteTime (line 37) | func (r Rate) byteTime(n int) time.Duration {
type Listener (line 44) | type Listener struct
method Accept (line 50) | func (ln *Listener) Accept() (net.Conn, error) {
type nErr (line 61) | type nErr struct
type writeReq (line 66) | type writeReq struct
type conn (line 72) | type conn struct
method start (line 82) | func (c *conn) start() {
method writeLoop (line 87) | func (c *conn) writeLoop() {
method Close (line 106) | func (c *conn) Close() error {
method Write (line 115) | func (c *conn) Write(p []byte) (n int, err error) {
method Read (line 129) | func (c *conn) Read(p []byte) (n int, err error) {
FILE: oauthutil/oauth.go
constant TitleBarRedirectURL (line 33) | TitleBarRedirectURL = "urn:ietf:wg:oauth:2.0:oob"
type TokenSource (line 42) | type TokenSource struct
method Token (line 81) | func (src TokenSource) Token() (*oauth2.Token, error) {
function cachedToken (line 59) | func cachedToken(cacheFile string) (*oauth2.Token, error) {
function NewRefreshTokenSource (line 115) | func NewRefreshTokenSource(config *oauth2.Config, refreshToken string) o...
FILE: osutil/osutil.go
function Executable (line 26) | func Executable() (string, error) {
FILE: readerutil/bufreaderat.go
function NewBufferingReaderAt (line 23) | func NewBufferingReaderAt(r io.Reader) io.ReaderAt {
type bufReaderAt (line 27) | type bufReaderAt struct
method ReadAt (line 32) | func (br *bufReaderAt) ReadAt(p []byte, off int64) (n int, err error) {
FILE: readerutil/bufreaderat_test.go
type trackingReader (line 21) | type trackingReader struct
method Read (line 27) | func (t *trackingReader) Read(p []byte) (n int, err error) {
function TestBufferingReaderAt (line 40) | func TestBufferingReaderAt(t *testing.T) {
FILE: readerutil/countingreader.go
type CountingReader (line 23) | type CountingReader struct
method Read (line 28) | func (cr CountingReader) Read(p []byte) (n int, err error) {
FILE: readerutil/fakeseeker.go
type fakeSeeker (line 28) | type fakeSeeker struct
method Seek (line 43) | func (fs *fakeSeeker) Seek(offset int64, whence int) (int64, error) {
method Read (line 62) | func (fs *fakeSeeker) Read(p []byte) (n int, err error) {
function NewFakeSeeker (line 39) | func NewFakeSeeker(r io.Reader, size int64) io.ReadSeeker {
FILE: readerutil/fakeseeker_test.go
function TestFakeSeeker (line 25) | func TestFakeSeeker(t *testing.T) {
FILE: readerutil/multireaderat.go
function NewMultiReaderAt (line 26) | func NewMultiReaderAt(parts ...SizeReaderAt) SizeReaderAt {
type offsetAndSource (line 39) | type offsetAndSource struct
type multiRA (line 44) | type multiRA struct
method Size (line 49) | func (m *multiRA) Size() int64 { return m.size }
method ReadAt (line 51) | func (m *multiRA) ReadAt(p []byte, off int64) (n int, err error) {
function ZeroSizeReaderAt (line 95) | func ZeroSizeReaderAt(size int64) SizeReaderAt {
type allZeros (line 99) | type allZeros struct
method ReadAt (line 101) | func (a allZeros) ReadAt(p []byte, off int64) (n int, err error) {
method Size (line 108) | func (a allZeros) Size() int64 { return a.n }
FILE: readerutil/multireaderat_test.go
function TestMultiReaderAt (line 26) | func TestMultiReaderAt(t *testing.T) {
FILE: readerutil/readersize.go
function Size (line 29) | func Size(r io.Reader) (size int64, ok bool) {
FILE: readerutil/readersize_test.go
constant text (line 27) | text = "HelloWorld"
type testSrc (line 29) | type testSrc struct
method run (line 35) | func (tsrc *testSrc) run(t *testing.T) {
function TestBytesBuffer (line 45) | func TestBytesBuffer(t *testing.T) {
function TestSeeker (line 51) | func TestSeeker(t *testing.T) {
FILE: readerutil/readerutil.go
type SizeReaderAt (line 28) | type SizeReaderAt interface
type ReadSeekCloser (line 34) | type ReadSeekCloser interface
type ReaderAtCloser (line 40) | type ReaderAtCloser interface
type varStatReader (line 48) | type varStatReader struct
method Read (line 59) | func (v *varStatReader) Read(p []byte) (int, error) {
function NewStatsReader (line 55) | func NewStatsReader(v *expvar.Int, r io.Reader) io.Reader {
type varStatReadSeeker (line 65) | type varStatReadSeeker struct
method Read (line 76) | func (v *varStatReadSeeker) Read(p []byte) (int, error) {
method Seek (line 82) | func (v *varStatReadSeeker) Seek(offset int64, whence int) (int64, err...
function NewStatsReadSeeker (line 72) | func NewStatsReadSeeker(v *expvar.Int, rs io.ReadSeeker) io.ReadSeeker {
FILE: readerutil/readerutil_test.go
function ExampleNewStatsReader (line 27) | func ExampleNewStatsReader() {
FILE: readerutil/singlereader/opener.go
type openFile (line 36) | type openFile struct
type openFileHandle (line 42) | type openFileHandle struct
method Close (line 47) | func (f *openFileHandle) Close() error {
function Open (line 71) | func Open(path string) (readerutil.ReaderAtCloser, error) {
FILE: readerutil/singlereader/opener_test.go
function TestOpenSingle (line 28) | func TestOpenSingle(t *testing.T) {
FILE: reflectutil/swapper.go
function Swapper (line 20) | func Swapper(slice any) func(i, j int) {
FILE: rollsum/rollsum.go
constant windowSize (line 28) | windowSize = 64
constant charOffset (line 29) | charOffset = 31
constant blobBits (line 31) | blobBits = 13
constant blobSize (line 32) | blobSize = 1 << blobBits
type RollSum (line 34) | type RollSum struct
method add (line 47) | func (rs *RollSum) add(drop, add uint32) {
method Roll (line 54) | func (rs *RollSum) Roll(ch byte) {
method OnSplit (line 63) | func (rs *RollSum) OnSplit() bool {
method OnSplitWithBits (line 69) | func (rs *RollSum) OnSplitWithBits(n uint32) bool {
method Bits (line 74) | func (rs *RollSum) Bits() int {
method Digest (line 79) | func (rs *RollSum) Digest() uint32 {
function New (line 40) | func New() *RollSum {
FILE: rollsum/rollsum_test.go
function TestSum (line 24) | func TestSum(t *testing.T) {
function BenchmarkRollsum (line 75) | func BenchmarkRollsum(b *testing.B) {
FILE: sort/example_interface_test.go
type Person (line 13) | type Person struct
method String (line 18) | func (p Person) String() string {
type ByAge (line 24) | type ByAge
method Len (line 26) | func (a ByAge) Len() int { return len(a) }
method Swap (line 27) | func (a ByAge) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
method Less (line 28) | func (a ByAge) Less(i, j int) bool { return a[i].Age < a[j].Age }
function ExampleSort (line 30) | func ExampleSort() {
FILE: sort/example_keys_test.go
type earthMass (line 13) | type earthMass
type au (line 14) | type au
type Planet (line 17) | type Planet struct
type By (line 24) | type By
method Sort (line 27) | func (by By) Sort(planets []Planet) {
type planetSorter (line 36) | type planetSorter struct
method Len (line 42) | func (s *planetSorter) Len() int {
method Swap (line 47) | func (s *planetSorter) Swap(i, j int) {
method Less (line 52) | func (s *planetSorter) Less(i, j int) bool {
function Example_sortKeys (line 64) | func Example_sortKeys() {
FILE: sort/example_multi_test.go
type Change (line 13) | type Change struct
type lessFunc (line 19) | type lessFunc
type multiSorter (line 22) | type multiSorter struct
method Sort (line 28) | func (ms *multiSorter) Sort(changes []Change) {
method Len (line 42) | func (ms *multiSorter) Len() int {
method Swap (line 47) | func (ms *multiSorter) Swap(i, j int) {
method Less (line 56) | func (ms *multiSorter) Less(i, j int) bool {
function OrderedBy (line 35) | func OrderedBy(less ...lessFunc) *multiSorter {
function Example_sortMultiKeys (line 92) | func Example_sortMultiKeys() {
FILE: sort/example_slice_test.go
function Example (line 13) | func Example() {
function ExampleSlice (line 30) | func ExampleSlice() {
FILE: sort/example_test.go
function ExampleInts (line 12) | func ExampleInts() {
function ExampleReverse (line 19) | func ExampleReverse() {
FILE: sort/example_wrapper_test.go
type Grams (line 12) | type Grams
method String (line 14) | func (g Grams) String() string { return fmt.Sprintf("%dg", int(g)) }
type Organ (line 16) | type Organ struct
type Organs (line 21) | type Organs
method Len (line 23) | func (s Organs) Len() int { return len(s) }
method Swap (line 24) | func (s Organs) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
type ByName (line 28) | type ByName struct
method Less (line 30) | func (s ByName) Less(i, j int) bool { return s.Organs[i].Name < s.Orga...
type ByWeight (line 34) | type ByWeight struct
method Less (line 36) | func (s ByWeight) Less(i, j int) bool { return s.Organs[i].Weight < s....
function Example_sortWrapper (line 38) | func Example_sortWrapper() {
function printOrgans (line 73) | func printOrgans(s []*Organ) {
FILE: sort/export_test.go
function Heapsort (line 7) | func Heapsort(data Interface) {
FILE: sort/genzfunc.go
function main (line 30) | func main() {
type visitFunc (line 100) | type visitFunc
method Visit (line 102) | func (f visitFunc) Visit(n ast.Node) ast.Visitor { return f(n) }
function rewriteCalls (line 104) | func rewriteCalls(n ast.Node) ast.Visitor {
function rewriteCall (line 112) | func rewriteCall(ce *ast.CallExpr) {
FILE: sort/search.go
function Search (line 59) | func Search(n int, f func(int) bool) int {
function SearchInts (line 83) | func SearchInts(a []int, x int) int {
function SearchFloat64s (line 92) | func SearchFloat64s(a []float64, x float64) int {
function SearchStrings (line 101) | func SearchStrings(a []string, x string) int {
method Search (line 106) | func (p IntSlice) Search(x int) int { return SearchInts(p, x) }
method Search (line 109) | func (p Float64Slice) Search(x float64) int { return SearchFloat64s(p, x) }
method Search (line 112) | func (p StringSlice) Search(x string) int { return SearchStrings(p, x) }
FILE: sort/search_test.go
function f (line 13) | func f(a []int, x int) func(int) bool {
function TestSearch (line 51) | func TestSearch(t *testing.T) {
function log2 (line 63) | func log2(x int) int {
function TestSearchEfficiency (line 73) | func TestSearchEfficiency(t *testing.T) {
function TestSearchWrappers (line 113) | func TestSearchWrappers(t *testing.T) {
function runSearchWrappers (line 121) | func runSearchWrappers() {
function TestSearchWrappersDontAlloc (line 130) | func TestSearchWrappersDontAlloc(t *testing.T) {
function BenchmarkSearchWrappers (line 143) | func BenchmarkSearchWrappers(b *testing.B) {
function TestSearchExhaustive (line 152) | func TestSearchExhaustive(t *testing.T) {
FILE: sort/sort.go
type Interface (line 33) | type Interface interface
type lessSwap (line 46) | type lessSwap struct
function MakeInterface (line 53) | func MakeInterface(length int, swap func(i, j int), less func(i, j int) ...
function SliceSorter (line 60) | func SliceSorter(slice interface{}, less func(i, j int) bool) Interface {
function Slice (line 67) | func Slice(slice interface{}, less func(i, j int) bool) {
type funcs (line 74) | type funcs struct
method Len (line 79) | func (f *funcs) Len() int { return f.length }
method Swap (line 80) | func (f *funcs) Swap(i, j int) { f.lessSwap.Swap(i, j) }
method Less (line 81) | func (f *funcs) Less(i, j int) bool { return f.lessSwap.Less(i, j) }
function insertionSort (line 84) | func insertionSort(data Interface, a, b int) {
function siftDown (line 94) | func siftDown(data Interface, lo, hi, first int) {
function heapSort (line 112) | func heapSort(data Interface, a, b int) {
function medianOfThree (line 133) | func medianOfThree(data Interface, m1, m0, m2 int) {
function swapRange (line 149) | func swapRange(data Interface, a, b, n int) {
function doPivot (line 155) | func doPivot(data Interface, lo, hi int) (midlo, midhi int) {
function quickSort (line 242) | func quickSort(data Interface, a, b, maxDepth int) {
function Sort (line 278) | func Sort(data Interface) {
function With (line 290) | func With(length int, swap func(i, j int), less func(i, j int) bool) {
function maxDepth (line 296) | func maxDepth(n int) int {
type reverse (line 304) | type reverse struct
method Less (line 311) | func (r reverse) Less(i, j int) bool {
function Reverse (line 316) | func Reverse(data Interface) Interface {
function IsSorted (line 321) | func IsSorted(data Interface) bool {
type IntSlice (line 334) | type IntSlice
method Len (line 336) | func (p IntSlice) Len() int { return len(p) }
method Less (line 337) | func (p IntSlice) Less(i, j int) bool { return p[i] < p[j] }
method Swap (line 338) | func (p IntSlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
method Sort (line 341) | func (p IntSlice) Sort() { Sort(p) }
type Float64Slice (line 344) | type Float64Slice
method Len (line 346) | func (p Float64Slice) Len() int { return len(p) }
method Less (line 347) | func (p Float64Slice) Less(i, j int) bool { return p[i] < p[j] || isNa...
method Swap (line 348) | func (p Float64Slice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
method Sort (line 356) | func (p Float64Slice) Sort() { Sort(p) }
function isNaN (line 351) | func isNaN(f float64) bool {
type StringSlice (line 359) | type StringSlice
method Len (line 361) | func (p StringSlice) Len() int { return len(p) }
method Less (line 362) | func (p StringSlice) Less(i, j int) bool { return p[i] < p[j] }
method Swap (line 363) | func (p StringSlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
method Sort (line 366) | func (p StringSlice) Sort() { Sort(p) }
function Ints (line 371) | func Ints(a []int) { Sort(IntSlice(a)) }
function Float64s (line 374) | func Float64s(a []float64) { Sort(Float64Slice(a)) }
function Strings (line 377) | func Strings(a []string) { Sort(StringSlice(a)) }
function IntsAreSorted (line 380) | func IntsAreSorted(a []int) bool { return IsSorted(IntSlice(a)) }
function Float64sAreSorted (line 383) | func Float64sAreSorted(a []float64) bool { return IsSorted(Float64Slice(...
function StringsAreSorted (line 386) | func StringsAreSorted(a []string) bool { return IsSorted(StringSlice(a)) }
function Stable (line 418) | func Stable(data Interface) {
function stable (line 426) | func stable(data Interface, n int) {
function symMerge (line 469) | func symMerge(data Interface, a, m, b int) {
function rotate (line 555) | func rotate(data Interface, a, m, b int) {
FILE: sort/sort_test.go
function TestSlice (line 23) | func TestSlice(t *testing.T) {
function TestSortIntSlice (line 32) | func TestSortIntSlice(t *testing.T) {
function TestSortFloat64Slice (line 42) | func TestSortFloat64Slice(t *testing.T) {
function TestSortStringSlice (line 52) | func TestSortStringSlice(t *testing.T) {
function TestInts (line 62) | func TestInts(t *testing.T) {
function TestFloat64s (line 71) | func TestFloat64s(t *testing.T) {
function TestStrings (line 80) | func TestStrings(t *testing.T) {
function TestStringsWithSwapper (line 89) | func TestStringsWithSwapper(t *testing.T) {
function TestSortLarge_Random (line 100) | func TestSortLarge_Random(t *testing.T) {
function TestReverseSortIntSlice (line 118) | func TestReverseSortIntSlice(t *testing.T) {
type nonDeterministicTestingData (line 135) | type nonDeterministicTestingData struct
method Len (line 139) | func (t *nonDeterministicTestingData) Len() int {
method Less (line 142) | func (t *nonDeterministicTestingData) Less(i, j int) bool {
method Swap (line 148) | func (t *nonDeterministicTestingData) Swap(i, j int) {
function TestNonDeterministicComparison (line 154) | func TestNonDeterministicComparison(t *testing.T) {
function BenchmarkSortString1K (line 172) | func BenchmarkSortString1K(b *testing.B) {
function BenchmarkSortString1K_With (line 188) | func BenchmarkSortString1K_With(b *testing.B) {
function BenchmarkSortString1K_WithSwapper (line 208) | func BenchmarkSortString1K_WithSwapper(b *testing.B) {
function BenchmarkStableString1K (line 226) | func BenchmarkStableString1K(b *testing.B) {
function BenchmarkSortInt1K (line 242) | func BenchmarkSortInt1K(b *testing.B) {
function BenchmarkStableInt1K (line 255) | func BenchmarkStableInt1K(b *testing.B) {
function BenchmarkStableInt1K_With (line 270) | func BenchmarkStableInt1K_With(b *testing.B) {
function BenchmarkStableInt1K_WithSwapper (line 289) | func BenchmarkStableInt1K_WithSwapper(b *testing.B) {
function BenchmarkSortInt64K (line 306) | func BenchmarkSortInt64K(b *testing.B) {
function BenchmarkStableInt64K (line 319) | func BenchmarkStableInt64K(b *testing.B) {
constant _Sawtooth (line 333) | _Sawtooth = iota
constant _Rand (line 334) | _Rand
constant _Stagger (line 335) | _Stagger
constant _Plateau (line 336) | _Plateau
constant _Shuffle (line 337) | _Shuffle
constant _NDist (line 338) | _NDist
constant _Copy (line 342) | _Copy = iota
constant _Reverse (line 343) | _Reverse
constant _ReverseFirstHalf (line 344) | _ReverseFirstHalf
constant _ReverseSecondHalf (line 345) | _ReverseSecondHalf
constant _Sorted (line 346) | _Sorted
constant _Dither (line 347) | _Dither
constant _NMode (line 348) | _NMode
type testingData (line 351) | type testingData struct
method Len (line 359) | func (d *testingData) Len() int { return len(d.data) }
method Less (line 360) | func (d *testingData) Less(i, j int) bool {
method Swap (line 364) | func (d *testingData) Swap(i, j int) {
function min (line 373) | func min(a, b int) int {
function lg (line 380) | func lg(n int) int {
function testBentleyMcIlroy (line 388) | func testBentleyMcIlroy(t *testing.T, sort func(Interface), maxswap func...
function TestSortBM (line 486) | func TestSortBM(t *testing.T) {
function TestHeapsortBM (line 490) | func TestHeapsortBM(t *testing.T) {
function TestStableBM (line 494) | func TestStableBM(t *testing.T) {
type adversaryTestingData (line 500) | type adversaryTestingData struct
method Len (line 506) | func (d *adversaryTestingData) Len() int { return len(d.data) }
method Less (line 508) | func (d *adversaryTestingData) Less(i, j int) bool {
method Swap (line 531) | func (d *adversaryTestingData) Swap(i, j int) {
function TestAdversary (line 535) | func TestAdversary(t *testing.T) {
function TestStableInts (line 546) | func TestStableInts(t *testing.T) {
type intPairs (line 554) | type intPairs
method Len (line 559) | func (d intPairs) Len() int { return len(d) }
method Less (line 560) | func (d intPairs) Less(i, j int) bool { return d[i].a < d[j].a }
method Swap (line 561) | func (d intPairs) Swap(i, j int) { d[i], d[j] = d[j], d[i] }
method initB (line 564) | func (d intPairs) initB() {
method inOrder (line 571) | func (d intPairs) inOrder() bool {
function TestStability (line 587) | func TestStability(t *testing.T) {
function countOps (line 636) | func countOps(t *testing.T, algo func(Interface), name string) {
function TestCountStableOps (line 659) | func TestCountStableOps(t *testing.T) { countOps(t, Stable, "Stable") }
function TestCountSortOps (line 660) | func TestCountSortOps(t *testing.T) { countOps(t, Sort, "Sort ") }
function bench (line 662) | func bench(b *testing.B, size int, algo func(Interface), name string) {
function BenchmarkSort1e2 (line 690) | func BenchmarkSort1e2(b *testing.B) { bench(b, 1e2, Sort, "Sort") }
function BenchmarkStable1e2 (line 691) | func BenchmarkStable1e2(b *testing.B) { bench(b, 1e2, Stable, "Stable") }
function BenchmarkSort1e4 (line 692) | func BenchmarkSort1e4(b *testing.B) { bench(b, 1e4, Sort, "Sort") }
function BenchmarkStable1e4 (line 693) | func BenchmarkStable1e4(b *testing.B) { bench(b, 1e4, Stable, "Stable") }
function BenchmarkSort1e6 (line 694) | func BenchmarkSort1e6(b *testing.B) { bench(b, 1e6, Sort, "Sort") }
function BenchmarkStable1e6 (line 695) | func BenchmarkStable1e6(b *testing.B) { bench(b, 1e6, Stable, "Stable") }
FILE: sort/zfuncversion.go
function insertionSort_func (line 10) | func insertionSort_func(data lessSwap, a, b int) {
function siftDown_func (line 19) | func siftDown_func(data lessSwap, lo, hi, first int) {
function heapSort_func (line 38) | func heapSort_func(data lessSwap, a, b int) {
function medianOfThree_func (line 52) | func medianOfThree_func(data lessSwap, m1, m0, m2 int) {
function swapRange_func (line 65) | func swapRange_func(data lessSwap, a, b, n int) {
function doPivot_func (line 72) | func doPivot_func(data lessSwap, lo, hi int) (midlo, midhi int) {
function quickSort_func (line 136) | func quickSort_func(data lessSwap, a, b, maxDepth int) {
function stable_func (line 163) | func stable_func(data lessSwap, n int) {
function symMerge_func (line 187) | func symMerge_func(data lessSwap, a, m, b int) {
function rotate_func (line 252) | func rotate_func(data lessSwap, a, m, b int) {
FILE: strutil/intern.go
function RegisterCommonString (line 24) | func RegisterCommonString(s ...string) {
function StringFromBytes (line 32) | func StringFromBytes(v []byte) string {
FILE: strutil/strconv.go
function ParseUintBytes (line 25) | func ParseUintBytes(s []byte, base int, bitSize int) (n uint64, err erro...
function cutoff64 (line 112) | func cutoff64(base int) uint64 {
FILE: strutil/strutil.go
function genSplit (line 29) | func genSplit(dst []string, s, sep string, sepSave, n int) []string {
function AppendSplitN (line 60) | func AppendSplitN(dst []string, s, sep string, n int) []string {
function equalFoldRune (line 67) | func equalFoldRune(sr, tr rune) bool {
function HasPrefixFold (line 98) | func HasPrefixFold(s, prefix string) bool {
function HasSuffixFold (line 121) | func HasSuffixFold(s, suffix string) bool {
function ContainsFold (line 148) | func ContainsFold(s, substr string) bool {
function IsPlausibleJSON (line 169) | func IsPlausibleJSON(s string) bool {
function isASCIIWhite (line 173) | func isASCIIWhite(b byte) bool { return b == ' ' || b == '\n' || b == '\...
function startsWithOpenBrace (line 175) | func startsWithOpenBrace(s string) bool {
function endsWithCloseBrace (line 189) | func endsWithCloseBrace(s string) bool {
FILE: strutil/strutil_test.go
function TestAppendSplitN (line 25) | func TestAppendSplitN(t *testing.T) {
function TestStringFromBytes (line 52) | func TestStringFromBytes(t *testing.T) {
function TestHasPrefixFold (line 61) | func TestHasPrefixFold(t *testing.T) {
function TestHasSuffixFold (line 91) | func TestHasSuffixFold(t *testing.T) {
function TestContainsFold (line 120) | func TestContainsFold(t *testing.T) {
function TestIsPlausibleJSON (line 174) | func TestIsPlausibleJSON(t *testing.T) {
function BenchmarkHasSuffixFoldToLower (line 198) | func BenchmarkHasSuffixFoldToLower(tb *testing.B) {
function BenchmarkHasSuffixFold (line 206) | func BenchmarkHasSuffixFold(tb *testing.B) {
function BenchmarkHasPrefixFoldToLower (line 215) | func BenchmarkHasPrefixFoldToLower(tb *testing.B) {
function BenchmarkHasPrefixFold (line 223) | func BenchmarkHasPrefixFold(tb *testing.B) {
FILE: syncutil/gate.go
type Gate (line 20) | type Gate struct
method Start (line 30) | func (g *Gate) Start() {
method Done (line 35) | func (g *Gate) Done() {
function NewGate (line 25) | func NewGate(max int) *Gate {
FILE: syncutil/group.go
type Group (line 23) | type Group struct
method Go (line 31) | func (g *Group) Go(fn func() error) {
method Wait (line 45) | func (g *Group) Wait() {
method Err (line 51) | func (g *Group) Err() error {
method Errs (line 61) | func (g *Group) Errs() []error {
FILE: syncutil/once.go
type Once (line 28) | type Once struct
method Do (line 45) | func (o *Once) Do(f func() error) error {
FILE: syncutil/once_test.go
function TestOnce (line 8) | func TestOnce(t *testing.T) {
function TestOnceErroring (line 33) | func TestOnceErroring(t *testing.T) {
FILE: syncutil/sem.go
type debugT (line 9) | type debugT
method Printf (line 13) | func (d debugT) Printf(format string, args ...interface{}) {
type Sem (line 21) | type Sem struct
method Acquire (line 39) | func (s *Sem) Acquire(n int64) error {
method Release (line 58) | func (s *Sem) Release(n int64) {
function NewSem (line 27) | func NewSem(max int64) *Sem {
FILE: syncutil/sem_test.go
function TestSem (line 9) | func TestSem(t *testing.T) {
function TestSemErr (line 28) | func TestSemErr(t *testing.T) {
FILE: syncutil/singleflight/singleflight.go
type call (line 24) | type call struct
type Group (line 32) | type Group struct
method Do (line 41) | func (g *Group) Do(key string, fn func() (interface{}, error)) (interf...
FILE: syncutil/singleflight/singleflight_test.go
function TestDo (line 28) | func TestDo(t *testing.T) {
function TestDoErr (line 41) | func TestDoErr(t *testing.T) {
function TestDoDupSuppress (line 55) | func TestDoDupSuppress(t *testing.T) {
FILE: syncutil/syncdebug/syncdebug.go
type RWMutexTracker (line 35) | type RWMutexTracker struct
method startLogger (line 96) | func (m *RWMutexTracker) startLogger() {
method Lock (line 117) | func (m *RWMutexTracker) Lock() {
method Unlock (line 133) | func (m *RWMutexTracker) Unlock() {
method RLock (line 142) | func (m *RWMutexTracker) RLock() {
method RUnlock (line 180) | func (m *RWMutexTracker) RUnlock() {
method Holder (line 194) | func (m *RWMutexTracker) Holder() string {
constant stackBufSize (line 51) | stackBufSize = 16 << 20
function getBuf (line 55) | func getBuf() []byte {
function putBuf (line 64) | func putBuf(b []byte) {
function GoroutineID (line 78) | func GoroutineID() int64 {
function stack (line 175) | func stack() []byte {
FILE: syncutil/syncdebug/syncdebug_test.go
function TestGoroutineID (line 21) | func TestGoroutineID(t *testing.T) {
FILE: testing/functest/functest.go
type Func (line 84) | type Func struct
method Case (line 160) | func (f *Func) Case(name string) *Case {
method In (line 166) | func (f *Func) In(args ...interface{}) *Case {
method Test (line 208) | func (f *Func) Test(t testing.TB, cases ...*Case) {
method checkCall (line 214) | func (f *Func) checkCall(in []reflect.Value) (out []reflect.Value, did...
method testCase (line 224) | func (f *Func) testCase(t testing.TB, c *Case) {
function New (line 98) | func New(f interface{}) *Func {
type Result (line 128) | type Result struct
type Case (line 151) | type Case struct
method In (line 171) | func (c *Case) In(args ...interface{}) *Case {
method Want (line 179) | func (c *Case) Want(result ...interface{}) *Case {
method Check (line 201) | func (c *Case) Check(checker func(Result) error) *Case {
method errorf (line 298) | func (c *Case) errorf(t testing.TB, format string, args ...interface{}) {
function formatRes (line 277) | func formatRes(res []interface{}) string {
function formatValues (line 289) | func formatValues(buf *bytes.Buffer, vals []interface{}) {
FILE: testing/functest/functest_test.go
type trec (line 10) | type trec struct
method Errorf (line 15) | func (t *trec) Errorf(format string, args ...interface{}) {
method Logf (line 21) | func (t *trec) Logf(format string, args ...interface{}) {
method String (line 27) | func (t *trec) String() string { return t.buf.String() }
function add (line 29) | func add(a, b int) int { return a + b }
function TestBasic (line 31) | func TestBasic(t *testing.T) {
function TestBasic_Strings (line 47) | func TestBasic_Strings(t *testing.T) {
function TestVariadic (line 63) | func TestVariadic(t *testing.T) {
function condPanic (line 92) | func condPanic(doPanic bool, panicValue interface{}) {
function TestPanic (line 98) | func TestPanic(t *testing.T) {
function TestName_AutoFunc (line 121) | func TestName_AutoFunc(t *testing.T) {
type SomeType (line 125) | type SomeType struct
method SomeMethod (line 127) | func (t *SomeType) SomeMethod(int) int { return 123 }
function TestName_AutoMethod (line 129) | func TestName_AutoMethod(t *testing.T) {
function testName (line 133) | func testName(t *testing.T, f *Func, want string) {
FILE: types/types.go
type Time3339 (line 42) | type Time3339
method String (line 49) | func (t Time3339) String() string {
method MarshalJSON (line 53) | func (t Time3339) MarshalJSON() ([]byte, error) {
method UnmarshalJSON (line 60) | func (t *Time3339) UnmarshalJSON(b []byte) error {
method Time (line 106) | func (t Time3339) Time() time.Time {
method IsAnyZero (line 111) | func (t *Time3339) IsAnyZero() bool {
function ParseTime3339OrZero (line 87) | func ParseTime3339OrZero(v string) Time3339 {
function ParseTime3339OrNil (line 95) | func ParseTime3339OrNil(v string) *Time3339 {
type ByTime (line 116) | type ByTime
method Len (line 118) | func (s ByTime) Len() int { return len(s) }
method Less (line 119) | func (s ByTime) Less(i, j int) bool { return s[i].Before(s[j]) }
method Swap (line 120) | func (s ByTime) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
function NewOnceCloser (line 124) | func NewOnceCloser(c io.Closer) io.Closer {
type onceCloser (line 128) | type onceCloser struct
method Close (line 133) | func (c *onceCloser) Close() error {
type CloseFunc (line 145) | type CloseFunc
method Close (line 147) | func (fn CloseFunc) Close() error { return fn() }
FILE: types/types_test.go
function TestTime3339 (line 26) | func TestTime3339(t *testing.T) {
function TestTime3339_Marshal (line 51) | func TestTime3339_Marshal(t *testing.T) {
function TestTime3339_empty (line 71) | func TestTime3339_empty(t *testing.T) {
FILE: wkfs/gcs/gcs.go
constant maxSize (line 48) | maxSize = 1 << 20
function init (line 50) | func init() {
type gcsFS (line 71) | type gcsFS struct
method parseName (line 83) | func (fs *gcsFS) parseName(name string) (bucket, fileName string, err ...
method Open (line 97) | func (fs *gcsFS) Open(name string) (wkfs.File, error) {
method Stat (line 127) | func (fs *gcsFS) Stat(name string) (os.FileInfo, error) { return fs.Ls...
method Lstat (line 128) | func (fs *gcsFS) Lstat(name string) (os.FileInfo, error) {
method MkdirAll (line 146) | func (fs *gcsFS) MkdirAll(path string, perm os.FileMode) error { retur...
method OpenFile (line 148) | func (fs *gcsFS) OpenFile(name string, flag int, perm os.FileMode) (wk...
method Remove (line 168) | func (fs *gcsFS) Remove(name string) error {
function registerBrokenFS (line 77) | func registerBrokenFS(err error) {
type statInfo (line 176) | type statInfo struct
method IsDir (line 183) | func (si *statInfo) IsDir() bool { return si.isDir }
method ModTime (line 184) | func (si *statInfo) ModTime() time.Time { return si.modtime }
method Mode (line 185) | func (si *statInfo) Mode() os.FileMode { return 0644 }
method Name (line 186) | func (si *statInfo) Name() string { return path.Base(si.name) }
method Size (line 187) | func (si *statInfo) Size() int64 { return si.size }
method Sys (line 188) | func (si *statInfo) Sys() interface{} { return nil }
type file (line 190) | type file struct
method Close (line 195) | func (*file) Close() error { return nil }
method Name (line 196) | func (f *file) Name() string { return path.Base(f.name) }
method Stat (line 197) | func (f *file) Stat() (os.FileInfo, error) {
FILE: wkfs/gcs/gcs_test.go
function TestWriteRead (line 35) | func TestWriteRead(t *testing.T) {
FILE: wkfs/wkfs.go
type File (line 37) | type File interface
type FileWriter (line 46) | type FileWriter interface
function Open (line 51) | func Open(name string) (File, error) { return fs(name).Ope...
function Stat (line 52) | func Stat(name string) (os.FileInfo, error) { return fs(name).Sta...
function Lstat (line 53) | func Lstat(name string) (os.FileInfo, error) { return fs(name).Lst...
function MkdirAll (line 54) | func MkdirAll(path string, perm os.FileMode) error { return fs(path).Mkd...
function OpenFile (line 55) | func OpenFile(name string, flag int, perm os.FileMode) (FileWriter, erro...
function Remove (line 58) | func Remove(name string) error { return fs(name).Remove(name) }
function Create (line 59) | func Create(name string) (FileWriter, error) {
function fs (line 65) | func fs(name string) FileSystem {
type osFS (line 74) | type osFS struct
method Open (line 76) | func (osFS) Open(name string) (File, error) { return os....
method Stat (line 77) | func (osFS) Stat(name string) (os.FileInfo, error) { return os....
method Lstat (line 78) | func (osFS) Lstat(name string) (os.FileInfo, error) { return os....
method MkdirAll (line 79) | func (osFS) MkdirAll(path string, perm os.FileMode) error { return os....
method OpenFile (line 80) | func (osFS) OpenFile(name string, flag int, perm os.FileMode) (FileWri...
method Remove (line 83) | func (osFS) Remove(name string) error { return os.Remove(name) }
type FileSystem (line 85) | type FileSystem interface
function RegisterFS (line 100) | func RegisterFS(prefix string, fs FileSystem) {
function WriteFile (line 113) | func WriteFile(filename string, data []byte, perm os.FileMode) error {
function ReadFile (line 128) | func ReadFile(filename string) ([]byte, error) {
FILE: writerutil/writerutil.go
type PrefixSuffixSaver (line 29) | type PrefixSuffixSaver struct
method Write (line 43) | func (w *PrefixSuffixSaver) Write(p []byte) (n int, err error) {
method fill (line 69) | func (w *PrefixSuffixSaver) fill(dst *[]byte, p []byte) (pRemain []byt...
method Bytes (line 82) | func (w *PrefixSuffixSaver) Bytes() []byte {
function minInt (line 100) | func minInt(a, b int) int {
FILE: writerutil/writerutil_test.go
function TestPrefixSuffixSaver (line 24) | func TestPrefixSuffixSaver(t *testing.T) {
FILE: xdgdir/example_test.go
function Example (line 26) | func Example() {
FILE: xdgdir/xdgdir.go
function init (line 41) | func init() {
type Dir (line 67) | type Dir struct
method String (line 92) | func (d Dir) String() string {
method Path (line 102) | func (d Dir) Path() string {
method path (line 123) | func (d Dir) path() string {
method SearchPaths (line 143) | func (d Dir) SearchPaths() []string {
method Open (line 174) | func (d Dir) Open(name string) (*os.File, error) {
method Create (line 204) | func (d Dir) Create(name string) (*os.File, error) {
function isValidPath (line 219) | func isValidPath(path string) bool {
FILE: xdgdir/xdgdir_test.go
function TestDir_Path (line 26) | func TestDir_Path(t *testing.T) {
function TestDir_SearchPaths (line 153) | func TestDir_SearchPaths(t *testing.T) {
function TestDir_Open (line 240) | func TestDir_Open(t *testing.T) {
function TestDir_Create (line 341) | func TestDir_Create(t *testing.T) {
function stringsEqual (line 415) | func stringsEqual(a, b []string) bool {
type tempDir (line 427) | type tempDir struct
method newFile (line 443) | func (td *tempDir) newFile(name string, data string) string {
method mkdir (line 464) | func (td *tempDir) mkdir(name string, perm os.FileMode) string {
method cleanup (line 473) | func (td *tempDir) cleanup() {
function newTempDir (line 432) | func newTempDir(t *testing.T) *tempDir {
type permCheck (line 480) | type permCheck struct
type env (line 485) | type env
method set (line 487) | func (e env) set() {
FILE: ziputil/ziputil.go
constant methodStore (line 44) | methodStore uint16 = 0
constant methodDeflate (line 45) | methodDeflate uint16 = 8
constant fileHeaderSignature (line 49) | fileHeaderSignature = 0x04034b50
constant directoryHeaderSignature (line 50) | directoryHeaderSignature = 0x02014b50
constant directoryEndSignature (line 51) | directoryEndSignature = 0x06054b50
constant directory64LocSignature (line 52) | directory64LocSignature = 0x07064b50
constant directory64EndSignature (line 53) | directory64EndSignature = 0x06064b50
constant dataDescriptorSignature (line 54) | dataDescriptorSignature = 0x08074b50
constant fileHeaderLen (line 55) | fileHeaderLen = 30
constant directoryHeaderLen (line 56) | directoryHeaderLen = 46
constant directoryEndLen (line 57) | directoryEndLen = 22
constant dataDescriptorLen (line 58) | dataDescriptorLen = 16
constant dataDescriptor64Len (line 59) | dataDescriptor64Len = 24
constant directory64LocLen (line 60) | directory64LocLen = 20
constant directory64EndLen (line 61) | directory64EndLen = 56
constant creatorFAT (line 64) | creatorFAT = 0
constant creatorUnix (line 65) | creatorUnix = 3
constant creatorNTFS (line 66) | creatorNTFS = 11
constant creatorVFAT (line 67) | creatorVFAT = 14
constant creatorMacOSX (line 68) | creatorMacOSX = 19
constant zipVersion20 (line 71) | zipVersion20 = 20
constant zipVersion45 (line 72) | zipVersion45 = 45
constant uint16max (line 75) | uint16max = (1 << 16) - 1
constant uint32max (line 76) | uint32max = (1 << 32) - 1
constant zip64ExtraID (line 87) | zip64ExtraID = 0x0001
constant ntfsExtraID (line 88) | ntfsExtraID = 0x000a
constant unixExtraID (line 89) | unixExtraID = 0x000d
constant extTimeExtraID (line 90) | extTimeExtraID = 0x5455
constant infoZipUnixExtraID (line 91) | infoZipUnixExtraID = 0x5855
function ZipTOCSize (line 111) | func ZipTOCSize(size int64, zipFooter []byte) (tocSize int64, ok bool) {
type Reader (line 135) | type Reader struct
function ParseTOC (line 150) | func ParseTOC(size int64, toc []byte) (*Reader, error) {
function readDirectoryEnd (line 192) | func readDirectoryEnd(r io.ReaderAt, size int64) (dir *directoryEnd, bas...
type FileHeader (line 275) | type FileHeader struct
method hasDataDesc (line 810) | func (h *FileHeader) hasDataDesc() bool {
function readDirectoryHeader (line 362) | func readDirectoryHeader(f *FileHeader, r io.Reader) error {
function findDirectory64End (line 536) | func findDirectory64End(r io.ReaderAt, directoryEndOffset int64) (int64,...
function readDirectory64End (line 561) | func readDirectory64End(r io.ReaderAt, offset int64, d *directoryEnd) (e...
function findSignatureInBlock (line 583) | func findSignatureInBlock(b []byte) int {
type readBuf (line 601) | type readBuf
method uint8 (line 603) | func (b *readBuf) uint8() uint8 {
method uint16 (line 609) | func (b *readBuf) uint16() uint16 {
method uint32 (line 615) | func (b *readBuf) uint32() uint32 {
method uint64 (line 621) | func (b *readBuf) uint64() uint64 {
method sub (line 627) | func (b *readBuf) sub(n int) readBuf {
type directoryEnd (line 633) | type directoryEnd struct
function msDosTimeToTime (line 647) | func msDosTimeToTime(dosDate, dosTime uint16) time.Time {
function detectUTF8 (line 667) | func detectUTF8(s string) (valid, require bool) {
function timeZone (line 689) | func timeZone(offset time.Duration) *time.Location {
constant openReadSlop (line 706) | openReadSlop = 512
function OpenWithReader (line 719) | func OpenWithReader(h *FileHeader, getRawReader func(offsize, size int64...
function newFlateReader (line 774) | func newFlateReader(r io.Reader) io.ReadCloser {
type pooledFlateReader (line 784) | type pooledFlateReader struct
method Read (line 789) | func (r *pooledFlateReader) Read(p []byte) (n int, err error) {
method Close (line 798) | func (r *pooledFlateReader) Close() error {
function skipHeader (line 817) | func skipHeader(r io.Reader) (variableLen int64, err error) {
type checksumReader (line 836) | type checksumReader struct
method setErr (line 846) | func (r *checksumReader) setErr(err error) error {
method Read (line 858) | func (r *checksumReader) Read(b []byte) (n int, err error) {
method Close (line 897) | func (r *checksumReader) Close() error {
function readDataDescriptor (line 909) | func readDataDescriptor(r io.Reader, f *FileHeader) error {
FILE: ziputil/ziputil_test.go
type testZip (line 18) | type testZip struct
method Len (line 24) | func (tz *testZip) Len() int { return len(tz.data) }
method Size (line 25) | func (tz *testZip) Size() int64 { return int64(len(tz.data)) }
function mkZip (line 27) | func mkZip(t testing.TB, name string, numFiles int) *testZip {
function TestZipTOCSize (line 47) | func TestZipTOCSize(t *testing.T) {
function goTestZips (line 145) | func goTestZips(t testing.TB) (baseNames []string) {
function TestGoTestZips (line 159) | func TestGoTestZips(t *testing.T) {
function TestOpenWithReader (line 199) | func TestOpenWithReader(t *testing.T) {
type closeTracker (line 265) | type closeTracker struct
method Close (line 270) | func (c *closeTracker) Close() error {
Condensed preview — 103 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (402K chars).
[
{
"path": ".gitignore",
"chars": 266,
"preview": "# Compiled Object files, Static and Dynamic libs (Shared Objects)\n*.o\n*.a\n*.so\n\n# Folders\n_obj\n_test\n\n# Architecture spe"
},
{
"path": ".travis.yml",
"chars": 213,
"preview": "language: go\narch:\n - amd64\n - ppc64le\n\ngo:\n - \"1.13.x\"\n - tip\n\ngo_import_path: go4.org\n\nbefore_install:\n - go "
},
{
"path": "AUTHORS",
"chars": 314,
"preview": "# This is the official list of go4 authors for copyright purposes.\n# This is distinct from the CONTRIBUTORS file, which "
},
{
"path": "LICENSE",
"chars": 11358,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 2398,
"preview": "# go4\n\n[](https://travis-ci.org/go4org/go4 \"Travis CI"
},
{
"path": "bytereplacer/bytereplacer.go",
"chars": 7300,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "bytereplacer/bytereplacer_test.go",
"chars": 11257,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "cloud/cloudlaunch/cloudlaunch.go",
"chars": 12424,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "cloud/google/gceutil/gceutil.go",
"chars": 3803,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "cloud/google/gcsutil/storage.go",
"chars": 5185,
"preview": "/*\nCopyright 2015 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "ctxutil/ctxutil.go",
"chars": 1461,
"preview": "/*\nCopyright 2015 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "errorutil/highlight.go",
"chars": 1611,
"preview": "/*\nCopyright 2011 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "fault/fault.go",
"chars": 1691,
"preview": "/*\nCopyright 2014 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "go.mod",
"chars": 2719,
"preview": "module go4.org\n\ngo 1.24.0\n\nrequire (\n\tcloud.google.com/go/compute/metadata v0.9.0\n\tcloud.google.com/go/storage v1.56.0\n\t"
},
{
"path": "go.sum",
"chars": 12251,
"preview": "cel.dev/expr v0.24.0 h1:56OvJKSH3hDGL0ml5uSxZmz3/3Pq4tJ+fb1unVLAFcY=\ncel.dev/expr v0.24.0/go.mod h1:hLPLo1W4QUmuYdA72RBX"
},
{
"path": "go4test/cloudlaunch/serve_on_cloud.go",
"chars": 2601,
"preview": "//go:build ignore\n\n/*\nCopyright 2016 The Go4 Authors.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\ny"
},
{
"path": "jsonconfig/eval.go",
"chars": 8844,
"preview": "/*\nCopyright 2011 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "jsonconfig/jsonconfig.go",
"chars": 6588,
"preview": "/*\nCopyright 2011 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "jsonconfig/jsonconfig_test.go",
"chars": 2754,
"preview": "/*\nCopyright 2011 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "jsonconfig/testdata/boolenv.json",
"chars": 361,
"preview": "{\n \"emptystr\": [\"_env\", \"${TEST_EMPTY}\", \"\"],\n \"def_false\": [\"_env\", \"${TEST_EMPTY}\", false],\n \"def_true\": [\"_env\", \"${T"
},
{
"path": "jsonconfig/testdata/include1.json",
"chars": 52,
"preview": "{\n \"two\": [\"_fileobj\", \"testdata/include2.json\"]\n}\n"
},
{
"path": "jsonconfig/testdata/include1bis.json",
"chars": 43,
"preview": "{\n \"two\": [\"_fileobj\", \"include2.json\"]\n}\n"
},
{
"path": "jsonconfig/testdata/include2.json",
"chars": 21,
"preview": "{\n \"key\": \"value\"\n}\n"
},
{
"path": "jsonconfig/testdata/listexpand.json",
"chars": 81,
"preview": "{\n \"list\": [\"foo\", [\"_env\", \"${TEST_BAR}\"]],\n \"str\": [\"_env\", \"${TEST_BAR}\"]\n}\n"
},
{
"path": "jsonconfig/testdata/loop1.json",
"chars": 49,
"preview": "{\n \"obj\": [\"_fileobj\", \"testdata/loop2.json\"]\n}\n"
},
{
"path": "jsonconfig/testdata/loop2.json",
"chars": 49,
"preview": "{\n \"obj\": [\"_fileobj\", \"testdata/loop1.json\"]\n}\n"
},
{
"path": "legal/legal.go",
"chars": 974,
"preview": "/*\nCopyright 2014 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "legal/legal_test.go",
"chars": 767,
"preview": "/*\nCopyright 2014 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "lock/.gitignore",
"chars": 3,
"preview": "*~\n"
},
{
"path": "lock/lock.go",
"chars": 4788,
"preview": "/*\nCopyright 2013 The Go Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this f"
},
{
"path": "lock/lock_plan9.go",
"chars": 1036,
"preview": "/*\nCopyright 2013 The Go Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this f"
},
{
"path": "lock/lock_sigzero.go",
"chars": 727,
"preview": "//go:build linux || darwin || freebsd || openbsd || netbsd || dragonfly || solaris\n\n/*\nCopyright 2013 The Go Authors\n\nLi"
},
{
"path": "lock/lock_test.go",
"chars": 4515,
"preview": "/*\nCopyright 2013 The Go Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this f"
},
{
"path": "lock/lock_unix.go",
"chars": 1420,
"preview": "//go:build linux || darwin || freebsd || openbsd || netbsd || dragonfly || solaris\n\n/*\nCopyright 2013 The Go Authors\n\nLi"
},
{
"path": "lock/lock_windows.go",
"chars": 1800,
"preview": "/*\nCopyright 2013 The Go Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this f"
},
{
"path": "media/heif/bmff/bmff.go",
"chars": 19036,
"preview": "/*\nCopyright 2018 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "media/heif/dumpheif/dumpheif.go",
"chars": 5315,
"preview": "/*\nCopyright 2018 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "media/heif/heif.go",
"chars": 7434,
"preview": "/*\nCopyright 2018 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "media/heif/heif_test.go",
"chars": 2498,
"preview": "package heif\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/rwcarlsen/goexif/exif\"\n\t\"github.com/rwcarlsen/goex"
},
{
"path": "must/must.go",
"chars": 1037,
"preview": "/*\nCopyright 2019 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "net/throttle/throttle.go",
"chars": 2833,
"preview": "/*\nCopyright 2012 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "oauthutil/oauth.go",
"chars": 4137,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "osutil/osutil.go",
"chars": 1002,
"preview": "/*\nCopyright 2015 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/bufreaderat.go",
"chars": 1241,
"preview": "/*\nCopyright 2018 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/bufreaderat_test.go",
"chars": 1956,
"preview": "/*\nCopyright 2018 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/countingreader.go",
"chars": 886,
"preview": "/*\nCopyright 2011 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/fakeseeker.go",
"chars": 1775,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "readerutil/fakeseeker_test.go",
"chars": 1896,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "readerutil/multireaderat.go",
"chars": 2406,
"preview": "/*\nCopyright 2016 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/multireaderat_test.go",
"chars": 1346,
"preview": "/*\nCopyright 2016 The Go4 Authors.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this"
},
{
"path": "readerutil/readersize.go",
"chars": 1460,
"preview": "/*\nCopyright 2012 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/readersize_test.go",
"chars": 1442,
"preview": "/*\nCopyright 2012 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/readerutil.go",
"chars": 2018,
"preview": "/*\nCopyright 2016 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/readerutil_test.go",
"chars": 1012,
"preview": "/*\nCopyright 2016 The Go4 Authors.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this"
},
{
"path": "readerutil/singlereader/opener.go",
"chars": 2610,
"preview": "/*\nCopyright 2013 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "readerutil/singlereader/opener_test.go",
"chars": 1692,
"preview": "/*\nCopyright 2013 The Go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "reflectutil/swapper.go",
"chars": 760,
"preview": "// Copyright 2016 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "rollsum/rollsum.go",
"chars": 2245,
"preview": "/*\nCopyright 2011 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "rollsum/rollsum_test.go",
"chars": 2173,
"preview": "/*\nCopyright 2011 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "sort/example_interface_test.go",
"chars": 909,
"preview": "// Copyright 2011 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/example_keys_test.go",
"chars": 2745,
"preview": "// Copyright 2013 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/example_multi_test.go",
"chars": 4104,
"preview": "// Copyright 2013 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/example_slice_test.go",
"chars": 985,
"preview": "// Copyright 2016 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/example_test.go",
"chars": 481,
"preview": "// Copyright 2011 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/example_wrapper_test.go",
"chars": 1672,
"preview": "// Copyright 2011 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/export_test.go",
"chars": 239,
"preview": "// Copyright 2011 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/genzfunc.go",
"chars": 2899,
"preview": "// Copyright 2016 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/search.go",
"chars": 4230,
"preview": "// Copyright 2010 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/search_test.go",
"chars": 4348,
"preview": "// Copyright 2010 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/sort.go",
"chars": 18310,
"preview": "// Copyright 2009 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/sort_test.go",
"chars": 15373,
"preview": "// Copyright 2009 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "sort/zfuncversion.go",
"chars": 4896,
"preview": "// DO NOT EDIT; AUTO-GENERATED from sort.go using genzfunc.go\n\n// Copyright 2016 The Go Authors. All rights reserved.\n//"
},
{
"path": "strutil/intern.go",
"chars": 1266,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "strutil/strconv.go",
"chars": 2289,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "strutil/strutil.go",
"chars": 4802,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "strutil/strutil_test.go",
"chars": 5991,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "syncutil/gate.go",
"chars": 1039,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "syncutil/group.go",
"chars": 1595,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "syncutil/once.go",
"chars": 1741,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "syncutil/once_test.go",
"chars": 949,
"preview": "package syncutil\n\nimport (\n\t\"errors\"\n\t\"testing\"\n)\n\nfunc TestOnce(t *testing.T) {\n\ttimesRan := 0\n\tf := func() error {\n\t\tt"
},
{
"path": "syncutil/sem.go",
"chars": 1547,
"preview": "package syncutil\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"sync\"\n)\n\ntype debugT bool\n\nvar debug = debugT(false)\n\nfunc (d debugT) Printf("
},
{
"path": "syncutil/sem_test.go",
"chars": 500,
"preview": "package syncutil_test\n\nimport (\n\t\"testing\"\n\n\t\"go4.org/syncutil\"\n)\n\nfunc TestSem(t *testing.T) {\n\ts := syncutil.NewSem(5)"
},
{
"path": "syncutil/singleflight/singleflight.go",
"chars": 1724,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "syncutil/singleflight/singleflight_test.go",
"chars": 1903,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "syncutil/syncdebug/syncdebug.go",
"chars": 4608,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "syncutil/syncdebug/syncdebug_test.go",
"chars": 827,
"preview": "/*\nCopyright 2013 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "syncutil/syncutil.go",
"chars": 678,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "testing/functest/functest.go",
"chars": 7861,
"preview": "/*\nCopyright 2016 The go4.org Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "testing/functest/functest_test.go",
"chars": 3054,
"preview": "package functest\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"testing\"\n)\n\n// trec is a testing.TB which logs Errorf calls to buf\ntype tre"
},
{
"path": "types/types.go",
"chars": 3521,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "types/types_test.go",
"chars": 2645,
"preview": "/*\nCopyright 2013 Google Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file"
},
{
"path": "wkfs/gcs/gcs.go",
"chars": 5251,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "wkfs/gcs/gcs_test.go",
"chars": 2343,
"preview": "/*\nCopyright 2015 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "wkfs/wkfs.go",
"chars": 4274,
"preview": "/*\nCopyright 2014 The Perkeep Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use t"
},
{
"path": "writerutil/writerutil.go",
"chars": 3174,
"preview": "/*\nCopyright 2016 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "writerutil/writerutil_test.go",
"chars": 1652,
"preview": "/*\nCopyright 2016 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "xdgdir/example_test.go",
"chars": 1418,
"preview": "/*\nCopyright 2017 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "xdgdir/xdgdir.go",
"chars": 5881,
"preview": "/*\nCopyright 2017 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "xdgdir/xdgdir_test.go",
"chars": 11879,
"preview": "/*\nCopyright 2017 The go4 Authors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "ziputil/ziputil.go",
"chars": 30074,
"preview": "// Copyright 2026 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "ziputil/ziputil_test.go",
"chars": 6504,
"preview": "package ziputil\n\nimport (\n\t\"archive/zip\"\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"reflect\"\n\t\"strings\"\n\t\"sync/atomi"
}
]
// ... and 2 more files (download for full content)
About this extraction
This page contains the full source code of the camlistore/go4 GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 103 files (355.6 KB), approximately 112.8k tokens, and a symbol index with 775 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.