Full Code of luke8086/retronews for AI

Repository: luke8086/retronews
Branch: main
Commit: b3fd323e9053
Files: 38
Total size: 91.4 KB

Directory structure:
gitextract_03mcvkkn/

├── .flake8
├── .gitignore
├── LICENSE
├── Makefile
├── README.md
├── pyproject.toml
├── retronews.py
├── tests/
│   ├── test_hn_01.html
│   ├── test_hn_01.out
│   ├── test_hn_02.html
│   ├── test_hn_02.out
│   ├── test_hn_03.html
│   ├── test_hn_03.out
│   ├── test_hn_04.html
│   ├── test_hn_04.out
│   ├── test_hn_05.html
│   ├── test_hn_05.out
│   ├── test_hn_06.html
│   ├── test_hn_06.out
│   ├── test_lb_01.html
│   ├── test_lb_01.out
│   ├── test_lb_02.html
│   ├── test_lb_02.out
│   ├── test_lb_03.html
│   ├── test_lb_03.out
│   ├── test_lb_04.html
│   ├── test_lb_04.out
│   ├── test_lb_05.html
│   ├── test_lb_05.out
│   ├── test_lb_06.html
│   ├── test_lb_06.out
│   ├── test_lb_07.html
│   ├── test_lb_07.out
│   ├── test_lb_08.html
│   ├── test_lb_08.out
│   ├── test_lb_09.html
│   └── test_lb_09.out
└── tests.py

================================================
FILE CONTENTS
================================================

================================================
FILE: .flake8
================================================
[flake8]
max-line-length = 120


================================================
FILE: .gitignore
================================================
/.mypy_cache
/.venv
/__pycache__
/cache/*
/tmp/*


================================================
FILE: LICENSE
================================================
                    GNU GENERAL PUBLIC LICENSE
                       Version 2, June 1991

 Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

                            Preamble

  The licenses for most software are designed to take away your
freedom to share and change it.  By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.)  You can apply it to
your programs, too.

  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.

  To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.

  For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have.  You must make sure that they, too, receive or can get the
source code.  And you must show them these terms so they know their
rights.

  We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.

  Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software.  If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.

  Finally, any free program is threatened constantly by software
patents.  We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary.  To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.

  The precise terms and conditions for copying, distribution and
modification follow.

                    GNU GENERAL PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License.  The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language.  (Hereinafter, translation is included without limitation in
the term "modification".)  Each licensee is addressed as "you".

Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope.  The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.

  1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.

You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.

  2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:

    a) You must cause the modified files to carry prominent notices
    stating that you changed the files and the date of any change.

    b) You must cause any work that you distribute or publish, that in
    whole or in part contains or is derived from the Program or any
    part thereof, to be licensed as a whole at no charge to all third
    parties under the terms of this License.

    c) If the modified program normally reads commands interactively
    when run, you must cause it, when started running for such
    interactive use in the most ordinary way, to print or display an
    announcement including an appropriate copyright notice and a
    notice that there is no warranty (or else, saying that you provide
    a warranty) and that users may redistribute the program under
    these conditions, and telling the user how to view a copy of this
    License.  (Exception: if the Program itself is interactive but
    does not normally print such an announcement, your work based on
    the Program is not required to print an announcement.)

These requirements apply to the modified work as a whole.  If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works.  But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.

In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.

  3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:

    a) Accompany it with the complete corresponding machine-readable
    source code, which must be distributed under the terms of Sections
    1 and 2 above on a medium customarily used for software interchange; or,

    b) Accompany it with a written offer, valid for at least three
    years, to give any third party, for a charge no more than your
    cost of physically performing source distribution, a complete
    machine-readable copy of the corresponding source code, to be
    distributed under the terms of Sections 1 and 2 above on a medium
    customarily used for software interchange; or,

    c) Accompany it with the information you received as to the offer
    to distribute corresponding source code.  (This alternative is
    allowed only for noncommercial distribution and only if you
    received the program in object code or executable form with such
    an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for
making modifications to it.  For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable.  However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.

If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.

  4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.

  5. You are not required to accept this License, since you have not
signed it.  However, nothing else grants you permission to modify or
distribute the Program or its derivative works.  These actions are
prohibited by law if you do not accept this License.  Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.

  6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions.  You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.

  7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License.  If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all.  For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.

It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices.  Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.

This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.

  8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded.  In such case, this License incorporates
the limitation as if written in the body of this License.

  9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time.  Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.

Each version is given a distinguishing version number.  If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation.  If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.

  10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission.  For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this.  Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.

                            NO WARRANTY

  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.

  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.

                     END OF TERMS AND CONDITIONS

            How to Apply These Terms to Your New Programs

  If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.

  To do so, attach the following notices to the program.  It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.

    <one line to give the program's name and a brief idea of what it does.>
    Copyright (C) <year>  <name of author>

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License along
    with this program; if not, write to the Free Software Foundation, Inc.,
    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:

    Gnomovision version 69, Copyright (C) year name of author
    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
    This is free software, and you are welcome to redistribute it
    under certain conditions; type `show c' for details.

The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License.  Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.

You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary.  Here is a sample; alter the names:

  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
  `Gnomovision' (which makes passes at compilers) written by James Hacker.

  <signature of Ty Coon>, 1 April 1989
  Ty Coon, President of Vice

This General Public License does not permit incorporating your program into
proprietary programs.  If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library.  If this is what you want to do, use the GNU Lesser General
Public License instead of this License.


================================================
FILE: Makefile
================================================
SCRIPTS = retronews.py tests.py

.PHONY: venv-check
venv-check:
ifeq ("$(VIRTUAL_ENV)","")
	@echo "Venv is not active, run make venv-build and \`make venv-activate\`"
	@echo
	exit 1
endif

venv-build:
	python3 -m venv .venv

venv-activate:
	@echo "source .venv/bin/activate"

install: venv-check
	pip install black isort flake8 mypy

lint: venv-check
	black $(SCRIPTS)
	isort $(SCRIPTS)
	flake8 $(SCRIPTS)
	mypy $(SCRIPTS)

test:
	python3 tests.py


================================================
FILE: README.md
================================================
# retronews

A Python script for browsing [Hacker News](https://news.ycombinator.com/)
and [Lobsters](https://lobste.rs/) discussions with a nostalgic interface emulating
classical usenet and mail readers, like slrn and mutt.

It was primarily written so I could highlight interesting threads and keep
track of read / unread messages (see [blog post](https://luke8086.dev/retronews.html)).
The UI showing one message at a time also encourages slower, more focused reading.

On Unix-like systems it only depends on Python 3.9. On Windows you may also need
to install [windows-curses](https://pypi.org/project/windows-curses/).

It doesn't require installation. You can run it simply with:

```bash
$ curl -LO https://raw.githubusercontent.com/luke8086/retronews/main/retronews.py
$ python3 ./retronews.py
```

Press `?` to see available keybindings.

<img src="screenshot.png" width="600" />

## Customization

To customize retronews without directly editing the script, you can put any valid
Python code in `~/.retronewsrc.py` (or other location specified with `--rcfile`) to
be executed on startup. For example:

```python
# Ignore type warnings
from typing import Any
retronews: Any

# Example: Custom key bindings
retronews.KEY_BINDINGS[ord('a')] = lambda app: retronews.cmd_prev(app)
retronews.KEY_BINDINGS[ord('z')] = lambda app: retronews.cmd_next(app)

# Example: Custom colors
retronews.COLORS['author'] = (retronews.curses.COLOR_RED, -1)
```

## Known issues and limitations

- The reader is read-only, there are no plans to support voting and posting
- Message formatting is not perfect, but works well enough most of the time
- Detecting if threads contain unread responses works by only checking their
  count, it's not reliable if any responses were deleted

## Why not an NNTP gateway?

NNTP doesn't support browsing threads by title (let alone paginated) and requesting
their messages on demand. Clients need to fetch metadata of all available messages
in all available threads in advance. Given the volume of messages on HN, synchronizing
them to the gateway is not practical. Even when attempted, some clients struggle
with the sheer number of messages in a single group.

## Related projects

- [HN Search @ Algolia](https://hn.algolia.com/about) - The underlying API used to retrieve messages
- [nntpit](https://github.com/taviso/nntpit) - An NNTP gateway to reddit.com
- [circumflex](https://github.com/bensadeh/circumflex) - Another, more advanced TUI for HN


================================================
FILE: pyproject.toml
================================================
[tool.isort]
profile = "black"

[tool.black]
line-length = 120


================================================
FILE: retronews.py
================================================
#!/usr/bin/env python3
#
# Copyright (c) luke8086
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as published by
# the Free Software Foundation.
#

import sys

if sys.version_info < (3, 9):
    sys.stderr.write("Python 3.9 or newer is required.\n")
    sys.exit(1)

import argparse
import curses
import curses.textpad
import dataclasses
import html.parser
import json
import logging
import os
import re
import sqlite3
import traceback
import unicodedata
import urllib.request
import webbrowser
from collections import defaultdict
from datetime import datetime
from functools import partial, reduce
from textwrap import wrap
from typing import (
    TYPE_CHECKING,
    Any,
    Callable,
    Generator,
    Literal,
    NewType,
    Optional,
    TypedDict,
    TypeVar,
    Union,
    cast,
    overload,
)

KEY_BINDINGS: dict[int, Callable[["AppState"], None]] = {
    ord("q"): lambda app: cmd_quit(app),
    ord("?"): lambda app: cmd_help(app),
    ord("\n"): lambda app: cmd_open(app),
    ord(" "): lambda app: cmd_open(app),
    ord("o"): lambda app: cmd_show_links(app),
    ord("x"): lambda app: cmd_close(app),
    ord("s"): lambda app: cmd_star(app),
    ord("S"): lambda app: cmd_star_thread(app),
    ord("u"): lambda app: cmd_set_unread(app),
    ord("D"): lambda app: cmd_dump(app),
    ord("r"): lambda app: cmd_toggle_raw_mode(app),
    ord("k"): lambda app: cmd_up(app),
    ord("j"): lambda app: cmd_down(app),
    ord("p"): lambda app: cmd_prev(app),
    ord("n"): lambda app: cmd_next(app),
    ord("N"): lambda app: cmd_next_unread(app),
    ord("P"): lambda app: cmd_parent(app),
    ord(";"): lambda app: cmd_mark_set(app),
    ord(","): lambda app: cmd_mark_jump(app),
    ord("R"): lambda app: cmd_reload_page(app),
    ord("<"): lambda app: cmd_load_prev_page(app),
    ord(">"): lambda app: cmd_load_next_page(app),
    ord("g"): lambda app: cmd_load_page(app),
    curses.KEY_UP: lambda app: cmd_prev(app),
    curses.KEY_DOWN: lambda app: cmd_next(app),
    curses.KEY_PPAGE: lambda app: cmd_page_up(app),
    curses.KEY_NPAGE: lambda app: cmd_page_down(app),
    curses.KEY_RESIZE: lambda app: cmd_resize(app),
} | {ord(str(i)): lambda app, i=i: cmd_load_tab(app, i) for i in range(1, 10)}

HELP_MENU = "q:Quit  ?:Help  p:Prev  n:Next  N:Next-Unread  j:Down  k:Up  x:Close  s:Star"

HELP_SCREEN = """\
  q                       Quit retronews
  UP, DOWN                Go up / down by one message / pager line
  PG UP, PG DOWN          Gp up / down by one page of messages / pager lines
  p, n                    Go to previous / next message
  N                       Go to next unread message
  P                       Go to parent message
  ; ,                     Set mark, jump to mark & swap (valid within thread)
  RETURN, SPACE           Open selected message
  x                       Close current message / thread
  o                       Select link and open in browser
  1 - 9                   Change group
  R                       Refresh current page
  < >                     Go to previous / next page
  g                       Go to specific page
  k j                     Scroll pager up / down by one line
  s                       Star / unstar current message
  S                       Star / unstar current thread
  u                       Mark current message as unread
  r                       Toggle raw HTML mode

See https://github.com/luke8086/retronews for more information."""

Color = Literal[
    "author",
    "code",
    "cursor",
    "date",
    "default",
    "empty_pager_line",
    "deleted_message_pager_line",
    "menu",
    "menu_active",
    "nested_quote",
    "quote",
    "starred_subject",
    "header_subject",
    "tree",
    "unread_comments",
    "url",
]

COLORS: dict[Color, tuple[int, int]] = {
    "author": (curses.COLOR_YELLOW, -1),
    "code": (curses.COLOR_GREEN, -1),
    "cursor": (curses.COLOR_BLACK, curses.COLOR_CYAN),
    "date": (curses.COLOR_CYAN, -1),
    "default": (curses.COLOR_WHITE, -1),
    "empty_pager_line": (curses.COLOR_GREEN, -1),
    "deleted_message_pager_line": (curses.COLOR_RED, -1),
    "menu": (curses.COLOR_GREEN, curses.COLOR_BLUE),
    "menu_active": (curses.COLOR_YELLOW, curses.COLOR_BLUE),
    "nested_quote": (curses.COLOR_CYAN, -1),
    "quote": (curses.COLOR_YELLOW, -1),
    "starred_subject": (curses.COLOR_CYAN, -1),
    "header_subject": (curses.COLOR_GREEN, -1),
    "tree": (curses.COLOR_RED, -1),
    "unread_comments": (curses.COLOR_GREEN, -1),
    "url": (curses.COLOR_MAGENTA, -1),
}

REQUEST_TIMEOUT = 10

# Recognize ">text", "> text", ">>text", ">> text", etc.
QUOTE_REX = re.compile(r"^(> ?)+")

# Recognize "[n] link", "[n]: link", "[n] - link", etc.
REFERENCE_REX = re.compile(r"^\[\d+\][ :-]*https?://[^ ]*$")

# Recognize http/https URLs
URL_REX = re.compile(r"(https?://[^\s\)\"<,]+[^\s\)\"<,\.])")

# Recognize HN message URLs
HN_URL_REX = re.compile(r"^https://news\.ycombinator\.com/item\?id=(\d+)$")

# FIXME: Use TypeAlias after migrating to Python 3.10
DB = NewType("DB", "sqlite3.Connection")

if TYPE_CHECKING:
    from _curses import _CursesWindow

    Window = _CursesWindow
else:
    Window = Any

T = TypeVar("T")


class ExitException(Exception):
    code: int
    message: str

    def __init__(self, code: int = 0, message: str = ""):
        self.code = code
        self.message = message

        super().__init__(message)


@dataclasses.dataclass(frozen=True)
class Provider:
    fetch_thread: Callable[[str], "Message"]
    fetch_threads_by_id: Callable[[list[str]], list["Message"]]


PROVIDERS: dict[str, Provider] = {
    "hn": Provider(
        fetch_thread=lambda msg_id: hn_fetch_thread(msg_id),
        fetch_threads_by_id=lambda msg_ids: hn_fetch_threads_by_id(msg_ids),
    ),
    "lb": Provider(
        fetch_thread=lambda msg_id: lb_fetch_thread(msg_id),
        fetch_threads_by_id=lambda msg_ids: [lb_fetch_thread(x) for x in msg_ids],
    ),
}


@dataclasses.dataclass(frozen=True)
class Group:
    label: str
    fetch: Callable[[DB, int], list["Message"]]
    page: int = 1


GROUP_TABS: list[Group] = [
    Group(label="Front HN", fetch=lambda db, page: hn_fetch_threads("news", page)),
    Group(label="New HN", fetch=lambda db, page: hn_fetch_new_threads(page)),
    Group(label="Ask HN", fetch=lambda db, page: hn_fetch_threads("ask", page)),
    Group(label="Show HN", fetch=lambda db, page: hn_fetch_threads("show", page)),
    Group(label="Front LB", fetch=lambda db, page: lb_fetch_threads("", page)),
    Group(label="New LB", fetch=lambda db, page: lb_fetch_threads("newest", page)),
    Group(label="Starred", fetch=lambda db, page: group_fetch_starred_threads(db, page)),
]


@dataclasses.dataclass
class MessageFlags:
    read: bool = False
    starred: bool = False


@dataclasses.dataclass
class Message:
    msg_id: str
    thread_id: str
    content_location: str
    date: datetime
    author: Optional[str]
    title: str
    body: Optional[str] = None
    lines: list[str] = dataclasses.field(default_factory=list)
    parent: Optional["Message"] = None
    children: Optional[list["Message"]] = None
    flags: MessageFlags = dataclasses.field(default_factory=MessageFlags)
    read_comments: int = 0
    total_comments: int = 0
    index_position: int = 0
    index_tree: str = ""

    @property
    def is_read(self) -> bool:
        return self.flags.read

    @property
    def is_shown_as_read(self) -> bool:
        # If the message is an unloaded thread, check if all comments are read
        return self.read_comments >= self.total_comments if self.is_thread and self.children is None else self.is_read

    @property
    def is_thread(self) -> bool:
        return self.msg_id == self.thread_id

    @property
    def is_deleted(self) -> bool:
        return self.author is None


@dataclasses.dataclass
class Layout:
    lines: int = 0
    cols: int = 0
    top_menu_row: int = 0
    index_start: int = 1
    index_height: int = 0
    middle_menu_row: Optional[int] = None
    pager_start: Optional[int] = None
    pager_height: Optional[int] = None
    bottom_menu_row: int = 0
    flash_menu_row: int = 0


@dataclasses.dataclass
class AppState:
    screen: Window
    db: DB
    group: Group
    ascii: bool = False
    monochrome: bool = False
    colors: dict[Color, int] = dataclasses.field(default_factory=dict)
    messages: list[Message] = dataclasses.field(default_factory=list)
    messages_by_id: dict[str, Message] = dataclasses.field(default_factory=dict)
    selected_message: Optional[Message] = None
    marked_message_id: Optional[str] = None
    layout: Layout = dataclasses.field(default_factory=Layout)
    pager_visible: bool = False
    pager_offset: int = 0
    raw_mode: bool = False
    flash: Optional[str] = None


class HNSearchHit(TypedDict):
    objectID: int
    author: Optional[str]
    title: str
    created_at_i: int
    story_text: Optional[str]
    url: Optional[str]
    num_comments: int


class HNEntry(TypedDict):
    author: Optional[str]
    # FIXME: Recursive declarations are not yet supported in TypedDicts
    children: list[Any]
    created_at_i: int
    id: int
    parent_id: Optional[int]
    text: Optional[str]
    title: Optional[str]
    url: Optional[str]


class LBThread(TypedDict):
    short_id: str
    short_id_url: str
    created_at: str
    title: str
    url: str
    description: str
    submitter_user: str
    comment_count: int
    comments: Optional[list[Any]]


class LBComment(TypedDict):
    short_id: str
    created_at: str
    url: str
    commenting_user: str
    parent_comment: Optional[str]


HTML_BLOCK_TAGS = set(("root", "p", "pre", "blockquote", "ul", "ol", "li", "hr"))
HTML_INLINE_TAGS = set(("code", "a", "em", "strong", "b", "br"))
HTML_KNOWN_TAGS = HTML_BLOCK_TAGS.union(HTML_INLINE_TAGS)
HTML_AUTOCLOSE_TAGS = set(("hr", "br"))


@dataclasses.dataclass
class HTMLNode:
    tag: str

    parent: Optional["HTMLNode"] = None

    prev_sibling: Optional["HTMLNode"] = None
    next_sibling: Optional["HTMLNode"] = None

    first_child: Optional["HTMLNode"] = None
    last_child: Optional["HTMLNode"] = None

    attrs: dict[str, Optional[str]] = dataclasses.field(default_factory=dict)
    text: str = ""
    pre: bool = False


class HTMLParser(html.parser.HTMLParser):
    root_node: HTMLNode
    current_node: HTMLNode
    pre_level = 0

    def __init__(self):
        super().__init__()

        self.root_node = self.current_node = HTMLNode(tag="root")

    def handle_data(self, data: str) -> None:
        if self.current_node.tag in HTML_AUTOCLOSE_TAGS:
            self.handle_endtag(self.current_node.tag)

        node = HTMLNode(tag="text", text=data, pre=self.pre_level > 0)
        html_node_append(self.current_node, node)

    def handle_starttag(self, tag: str, attrs: list[tuple[str, Optional[str]]]) -> None:
        if self.current_node.tag in HTML_AUTOCLOSE_TAGS:
            self.handle_endtag(self.current_node.tag)

        if tag not in HTML_KNOWN_TAGS:
            return

        if tag == "pre":
            self.pre_level += 1

        node = HTMLNode(tag=tag, attrs=dict(attrs), pre=self.pre_level > 0)
        html_node_append(self.current_node, node)
        self.current_node = node

    def handle_endtag(self, tag: str) -> None:
        if tag not in HTML_KNOWN_TAGS:
            return

        if tag == "pre":
            self.pre_level = max(0, self.pre_level - 1)

        while True:
            node = self.current_node

            if node.parent is None:
                break

            self.current_node = node.parent

            if node.tag == tag:
                break


def text_wrap(text: str, width=70) -> str:
    if len(text) == 0:
        # Preserve empty lines
        return ""

    if text.startswith("  "):
        # Preserve code indentation
        return text

    if REFERENCE_REX.match(text):
        # Keep reference numbers with long links in the same line
        return text

    indent = ""

    if (match := QUOTE_REX.match(text)) is not None:
        # Preserve quotation symbols in subsequent lines
        indent = match[0]

    lines = wrap(text, width, subsequent_indent=indent, break_on_hyphens=False, break_long_words=False)
    lines = [line.rstrip() for line in lines]

    return "\n".join(lines)


def text_clean(text: str, ascii: bool = False) -> str:
    """Cleanup text for rendering, currently only removes non-ascii characters in ascii mode"""

    if ascii:
        text = text.encode("ascii", "replace").decode("ascii")

    return text


def text_unindent(text: str) -> str:
    lines = text.split("\n")

    while all(line.startswith(" ") or line == "" for line in lines):
        lines = [line[1:] for line in lines]

    return "\n".join(lines)


def text_sanitize(text: Optional[str]) -> str:
    # For safety, remove any control characters except for \n and \t
    # At least on HN some messages contain \x00 characters

    text = text or ""
    allowed_cc = set(("\n", "\t"))
    chars = (c for c in text if c in allowed_cc or unicodedata.category(c) != "Cc")
    text = "".join(chars)

    return text


def text_split_urls(text: str) -> list[str]:
    return [p for p in URL_REX.split(text) if p != ""]


def html_node_children(parent: HTMLNode) -> list[HTMLNode]:
    ret: list[HTMLNode] = []
    node = parent.first_child

    while node is not None:
        ret.append(node)
        node = node.next_sibling

    return ret


def html_node_append(parent: HTMLNode, child: HTMLNode) -> None:
    if parent.first_child is None:
        parent.first_child = child

    if parent.last_child is not None:
        parent.last_child.next_sibling = child
        child.prev_sibling = parent.last_child

    parent.last_child = child
    child.parent = parent


def html_node_unlink(node: HTMLNode) -> None:
    if node.prev_sibling:
        node.prev_sibling.next_sibling = node.next_sibling

    if node.next_sibling:
        node.next_sibling.prev_sibling = node.prev_sibling

    if node.parent and node.parent.first_child is node:
        node.parent.first_child = node.next_sibling

    if node.parent and node.parent.last_child is node:
        node.parent.last_child = node.prev_sibling

    node.parent = node.prev_sibling = node.next_sibling = None


def html_node_dump(node: HTMLNode) -> str:
    lines = []
    lines.append(f"{node.tag} {repr(node.attrs)}")

    for child in html_node_children(node):
        if child.tag == "text":
            lines.append("  text " + repr(child.text))
        else:
            lines += ["  " + line for line in html_node_dump(child).split("\n")]

    return "\n".join(lines)


def html_node_trim_whitespace(node: HTMLNode) -> None:
    if node.pre:
        node.text = node.text.rstrip("\r\n\t ")
        return

    text = node.text.strip("\r\n\t ")
    text = re.sub(r"[\r\n\t ]+", " ", text)
    text = text.replace("\x00", "\n")
    text = re.sub(r" *\n *", "\n", text)

    node.text = text


def html_node_process_inline(node: HTMLNode, inline=False) -> str:
    """Traverse tree flattening all inline nodes into text nodes"""

    if node.tag not in HTML_BLOCK_TAGS:
        # Never disable inline if already enabled
        inline = True

    text = "".join(html_node_process_inline(c, inline) for c in html_node_children(node))

    if not inline:
        return ""

    if node.tag == "text":
        text = node.text
    elif node.tag == "br":
        text = "\x00"
    elif node.tag == "em" or node.tag == "i":
        text = f"/{text}/"
    elif node.tag == "strong" or node.tag == "b":
        text = f"*{text}*"
    elif node.tag == "code" and not node.pre:
        text = f"`{text}`"
    elif node.tag == "a":
        href = node.attrs.get("href", "") or ""
        if text.endswith("...") and href.startswith(text[:-3]):
            # Workaround for link formatting on HN
            text = href
        elif text != href:
            text = f"{text} {href}"

    node.tag = "text"
    node.text = text
    node.first_child = node.last_child = None

    return text


def html_node_process_text(node: HTMLNode):
    """Traverse tree merging, trimming and pruning text nodes"""

    for child in html_node_children(node):
        html_node_process_text(child)

    # Merge adjacent text nodes
    for child in html_node_children(node):
        prev = child.prev_sibling
        if child.tag == "text" and prev is not None and prev.tag == "text" and child.pre == prev.pre:
            child.text = prev.text + child.text
            html_node_unlink(prev)

    # Trim whitespace from text nodes
    for child in html_node_children(node):
        if child.tag == "text":
            html_node_trim_whitespace(child)

    # Remove empty text nodes
    for child in html_node_children(node):
        if child.tag == "text" and child.text == "":
            html_node_unlink(child)


def html_node_render_block(node: HTMLNode, width=70) -> str:
    if node.tag == "blockquote" or node.tag == "li" or node.tag == "pre":
        width -= 2

    parts = []

    if node.tag == "hr":
        parts.append("-" * width)

    for child in html_node_children(node):
        if child.tag == "text" and child.pre:
            parts.append(child.text)

        elif child.tag == "text" and not child.pre:
            subparts = [text_wrap(p, width) for p in child.text.split("\n")]
            parts.append("\n".join(subparts))

        else:
            parts.append(html_node_render_block(child, width))

        if child.next_sibling is not None and child.tag != "li":
            parts.append("")

    text = "\n".join(parts)

    if node.tag == "blockquote":
        text = "\n".join((">" if line.startswith("> ") else "> ") + line for line in text.split("\n"))
    elif node.tag == "pre":
        text = text_unindent(text)
        text = "\n".join("| " + line for line in text.split("\n"))
    elif node.tag == "li":
        lines = text.split("\n")
        lines = [("- " + lines[0]).rstrip()] + [("  " + line).rstrip() for line in lines[1:]]
        text = "\n".join(lines)

    return text


def html_render(html: str) -> str:
    # This renderer works well for HN messages because their markup is simple, and it can do
    # some custom optimizations, like expanding ellipsis-shortened links, preserving quote
    # symbols in wrapped lines, and preventing references with long urls from being broken
    # into separate lines. For other backends it may make more sense to use an external app
    # (links, w3m, etc)

    html = text_sanitize(html)

    parser = HTMLParser()
    parser.feed(html)
    parser.close()

    node = parser.root_node

    log_sep = "\n" + "-" * 80 + "\n"
    logging.debug(f"Initial HTML tree{log_sep}{html_node_dump(node)}{log_sep}")

    html_node_process_inline(node)
    html_node_process_text(node)

    logging.debug(f"Processed HTML tree{log_sep}{html_node_dump(node)}{log_sep}")

    return html_node_render_block(node)


def fetch(url: str) -> str:
    logging.debug(f"Fetching '{url}'...")

    headers = {"User-Agent": "retronews"}
    req = urllib.request.Request(url, headers=headers)
    resp = urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT).read().decode()

    return resp


@overload
def list_get(lst: list[T], index: int, default: T) -> T: ...


@overload
def list_get(lst: list[T], index: int, default: Optional[T] = None) -> Optional[T]: ...


def list_get(lst, index, default=None):
    return lst[index] if 0 <= index < len(lst) else default


def list_chunk(lst: list[T], size: int) -> list[list[T]]:
    # Flake8 conflicts with Black here - https://github.com/PyCQA/pycodestyle/issues/373
    return [lst[i : i + size] for i in range(0, len(lst), size)]  # noqa: E203


def cmd_quit(_: AppState):
    raise ExitException()


def cmd_help(app: AppState):
    app_show_help_screen(app)


def cmd_show_links(app: AppState):
    app_show_links_screen(app)


def cmd_up(app: AppState) -> None:
    cmd_pager_up(app) if app.pager_visible else cmd_prev(app)


def cmd_down(app: AppState) -> None:
    cmd_pager_down(app) if app.pager_visible else cmd_next(app)


def cmd_prev(app: AppState) -> None:
    pos = app.selected_message.index_position - 1 if app.selected_message else 0
    app_select_message(app, list_get(app.messages, pos, app.selected_message))


def cmd_next(app: AppState) -> None:
    pos = app.selected_message.index_position + 1 if app.selected_message else 0
    app_select_message(app, list_get(app.messages, pos, app.selected_message))


def cmd_next_unread(app: AppState) -> None:
    pos = app.selected_message.index_position + 1 if app.selected_message else 0
    message = next((msg for msg in app.messages[pos:] if not msg.is_shown_as_read), None)
    if message is not None:
        app_select_message(app, message)


def cmd_next_sibling(app: AppState) -> None:
    if (msg := app.selected_message) and (parent_msg := msg.parent) and parent_msg.children:
        # find the next sibling
        try:
            idx = parent_msg.children.index(msg)
            if idx < len(parent_msg.children):
                app_select_message(app, parent_msg.children[idx + 1])
        except IndexError:
            pass


def cmd_prev_sibling(app: AppState) -> None:
    if (msg := app.selected_message) and (parent_msg := msg.parent) and parent_msg.children:
        # find the previous sibling
        try:
            idx = parent_msg.children.index(msg)
            if idx > 0:
                app_select_message(app, parent_msg.children[idx - 1])
        except IndexError:
            pass


def cmd_mark_thread_as_read(app: AppState) -> None:
    # recursively mark us and all children as read
    # then jump to the next sibling
    def iterate(message: Message):
        if message.children:
            for child in message.children:
                child.flags.read = True
                db_save_message(app.db, child)
                iterate(child)

    if (msg := app.selected_message) is not None:
        iterate(msg)
        # jump to the next sibling
        cmd_next_sibling(app)


def cmd_parent(app: AppState) -> None:
    if (msg := app.selected_message) is not None and (parent_msg := msg.parent) is not None:
        app_select_message(app, parent_msg)


def cmd_mark_set(app: AppState) -> None:
    if (msg := app.selected_message) is not None:
        app.marked_message_id = msg.msg_id
    app_show_flash(app, "Mark set")


def cmd_mark_jump(app: AppState) -> None:
    marked_msg = app.messages_by_id.get(app.marked_message_id) if app.marked_message_id else None
    cmd_mark_set(app)
    if marked_msg is not None:
        app_select_message(app, marked_msg)
    app_show_flash(app, "Mark swapped")


def cmd_pager_up(app: AppState) -> None:
    app.pager_offset = max(0, app.pager_offset - 1)


def cmd_pager_down(app: AppState) -> None:
    if app.selected_message is not None and app.layout.pager_height is not None:
        app.pager_offset = min(app.pager_offset + 1, max(0, len(app.selected_message.lines) - app.layout.pager_height))


def cmd_page_up(app: AppState) -> None:
    cmd_pager_page_up(app) if app.pager_visible else cmd_index_page_up(app)


def cmd_page_down(app: AppState) -> None:
    cmd_pager_page_down(app) if app.pager_visible else cmd_index_page_down(app)


def cmd_index_page_up(app: AppState) -> None:
    pos = app.selected_message.index_position - app.layout.index_height if app.selected_message else 0
    pos = max(pos, 0)
    app_select_message(app, list_get(app.messages, pos, app.selected_message))


def cmd_index_page_down(app: AppState) -> None:
    pos = app.selected_message.index_position + app.layout.index_height if app.selected_message else 0
    pos = min(pos, len(app.messages) - 1)
    app_select_message(app, list_get(app.messages, pos, app.selected_message))


def cmd_pager_page_up(app: AppState) -> None:
    if app.layout.pager_height is not None:
        app.pager_offset = max(0, app.pager_offset - app.layout.pager_height)


def cmd_pager_page_down(app: AppState) -> None:
    if (message := app.selected_message) is not None and (pager_height := app.layout.pager_height) is not None:
        app.pager_offset = min(app.pager_offset + pager_height, max(0, len(message.lines) - pager_height))


def cmd_load_tab(app: AppState, tab: int) -> None:
    if group := list_get(GROUP_TABS, tab - 1):
        app_load_group(app, group)


def cmd_reload_page(app: AppState) -> None:
    app_load_group(app, app.group)


def cmd_load_prev_page(app: AppState) -> None:
    app_load_group(app, group_advance_page(app.group, -1))


def cmd_load_next_page(app: AppState) -> None:
    app_load_group(app, group_advance_page(app.group, 1))


def cmd_load_page(app: AppState) -> None:
    user_input = app_prompt(app, "Go to page (empty to cancel): ")

    if user_input.isnumeric() and (page := int(user_input)) >= 1:
        app_load_group(app, group_set_page(app.group, page))
    elif len(user_input) > 0:
        app_show_flash(app, "Invalid page number")


def cmd_open(app: AppState) -> None:
    if (msg := app.selected_message) is None:
        return

    if msg.is_thread:
        app_open_thread(app, msg)
    else:
        app_select_message(app, msg, show_pager=True)


def cmd_close(app: AppState) -> None:
    if app.pager_visible:
        app.pager_visible = False
    else:
        app_close_thread(app)


def cmd_star(app: AppState) -> None:
    if (msg := app.selected_message) is not None:
        msg.flags.starred = not msg.flags.starred
        db_save_message(app.db, msg)
        cmd_next(app)


def cmd_star_thread(app: AppState) -> None:
    if (msg := app.selected_message) is None:
        return

    if (thread_msg := app.messages_by_id.get(msg.thread_id)) is None:
        return

    thread_msg.flags.starred = not thread_msg.flags.starred
    db_save_message(app.db, thread_msg)
    cmd_next(app)


def cmd_set_unread(app: AppState) -> None:
    if (msg := app.selected_message) is not None:
        msg.flags.read = False
        db_save_message(app.db, msg)
        cmd_next(app)


def cmd_dump(app: AppState) -> None:
    if (msg := app.selected_message) is None:
        return

    filename = f"{msg.msg_id}.html"

    with open(filename, "w") as fp:
        fp.write(msg.body or "")

    app_show_flash(app, f"Message body dumped to {filename}")


def cmd_toggle_raw_mode(app: AppState) -> None:
    app.raw_mode = not app.raw_mode
    app_select_message(app, app.selected_message)


def cmd_resize(app: AppState) -> None:
    app_refresh_message(app)


def cmd_unknown(app: AppState) -> None:
    app.flash = "Unknown key"


def db_init(path: str) -> DB:
    path = os.path.expanduser(path)
    create_table_sql = """
        CREATE TABLE IF NOT EXISTS messages (
            msg_id TEXT NOT NULL PRIMARY KEY,
            thread_id TEXT NOT NULL,
            date INTEGER NOT NULL,
            flags JSON NOT NULL
        );

        CREATE INDEX IF NOT EXISTS messages_starred_date ON messages (
            JSON_EXTRACT(flags, '$.starred'),
            date
        );
    """

    db = sqlite3.connect(path)
    db.row_factory = sqlite3.Row
    db.executescript(create_table_sql)
    db.commit()

    return cast(DB, db)


def db_save_message(db: DB, message: Message) -> None:
    sql = """INSERT OR REPLACE INTO messages (msg_id, thread_id, date, flags) VALUES (?, ?, ?, ?)"""
    date = int(message.date.timestamp())
    flags_json = json.dumps(dataclasses.asdict(message.flags))
    db.execute(sql, (message.msg_id, message.thread_id, date, flags_json))
    db.commit()


def db_load_message_flags(db: DB, messages_by_id: dict[str, Message]) -> None:
    message_ids = list(messages_by_id.keys())
    sql = f"SELECT * FROM messages WHERE msg_id IN ({','.join('?' for _ in message_ids)})"

    for row in db.execute(sql, message_ids):
        flags = json.loads(row["flags"])
        messages_by_id[row["msg_id"]].flags = MessageFlags(**flags)


def db_load_read_comments(db: DB, messages_by_id: dict[str, Message]) -> None:
    threads_by_id = {msg.msg_id: msg for msg in messages_by_id.values() if msg.is_thread}
    thread_ids = list(threads_by_id.keys())

    sql = f"""
        SELECT thread_id, COUNT(*) AS count
        FROM messages
        WHERE thread_id IN ({','.join('?' for _ in thread_ids)}) AND JSON_EXTRACT(flags, '$.read')
        GROUP BY thread_id
    """

    for row in db.execute(sql, thread_ids):
        threads_by_id[row["thread_id"]].read_comments = row["count"]


def db_load_starred_thread_ids(db: DB, page: int = 1) -> list[str]:
    page_size = 30
    offset = (page - 1) * page_size
    sql = """
        SELECT thread_id
        FROM messages
        WHERE JSON_EXTRACT(flags, '$.starred')
        GROUP BY thread_id
        ORDER BY date DESC
        LIMIT ?
        OFFSET ?
    """

    return [row["thread_id"] for row in db.execute(sql, (page_size, offset))]


def msg_flatten_thread(
    msg: Message, prefix: str = "", is_last_child: bool = False, ascii: bool = False
) -> Generator[Message, None, None]:
    blcorner = "'-" if ascii else "└─"
    ltee = "|-" if ascii else "├─"
    vline = "| " if ascii else "│ "

    msg.index_tree = "" if msg.is_thread else f"{prefix}{blcorner if is_last_child else ltee}> "
    yield msg

    children = msg.children or []

    child_count = len(children)
    child_prefix = "" if msg.is_thread else f"{prefix}{'  ' if is_last_child else vline}"

    for i, child_node in enumerate(children):
        child_is_last = i == child_count - 1
        for child in msg_flatten_thread(child_node, prefix=child_prefix, is_last_child=child_is_last, ascii=ascii):
            yield child


def msg_build_raw_lines(msg: Message) -> list[str]:
    text = text_sanitize(msg.body)

    # Unescape selected entities for better readability
    repl = {"&#x2F;": "/", "&#x27;": "'", "&quot;": '"'}
    for k, v in repl.items():
        text = text.replace(k, v)

    return reduce(lambda acc, line: acc + wrap(line, width=120, replace_whitespace=False), text.split("\n"), [])


def msg_build_lines(msg: Message) -> list[str]:
    lines = [
        f"Content-Location: {msg.content_location}",
        f"Date: {msg.date.strftime('%Y-%m-%d %H:%M')}",
        f"From: {msg.author or '<unknown>'}",
        f"Subject: {msg.title}",
        "",
    ]

    lines += html_render(msg.body or "").split("\n") if not msg.is_deleted else ["<deleted>"]

    return lines


def msg_unload(msg: Message) -> Message:
    msg.children = None
    msg.body = None
    return msg


def hn_parse_search_hit(hit: HNSearchHit) -> Message:
    return Message(
        msg_id=f"{hit['objectID']}@hn",
        thread_id=f"{hit['objectID']}@hn",
        content_location=f"https://news.ycombinator.com/item?id={hit['objectID']}",
        date=datetime.fromtimestamp(hit["created_at_i"]),
        author=hit["author"],
        title=html.unescape(hit["title"]),
        total_comments=(hit["num_comments"] or 0) + 1,
    )


def hn_parse_entry(entry: HNEntry, thread_id: str = "", parent: Optional[Message] = None) -> Message:
    thread_id = thread_id or str(entry["id"])

    my_title = html.unescape(entry["title"]) if entry["title"] else None

    parent_title = parent.title if parent else ""
    parent_title = parent_title if parent_title.startswith("Re: ") else f"Re: {parent_title}"

    body = f"<p>{entry['url']}</p>" if entry["url"] else ""
    body = f"{body}{entry['text']}" if entry["text"] else body

    msg = Message(
        msg_id=f"{entry['id']}@hn",
        thread_id=f"{thread_id}@hn",
        content_location=f"https://news.ycombinator.com/item?id={entry['id']}",
        date=datetime.fromtimestamp(entry["created_at_i"]),
        author=entry["author"],
        title=my_title or parent_title,
        body=body,
        parent=parent,
    )

    msg.children = [hn_parse_entry(child, thread_id, msg) for child in entry["children"]]

    return msg


def hn_fetch_threads_by_id(thread_ids: list[str]) -> list[Message]:
    story_tags = ",".join(f"story_{x}" for x in thread_ids)
    url = f"https://hn.algolia.com/api/v1/search_by_date?hitsPerPage={len(thread_ids)}&tags=story,({story_tags})"
    hits = json.loads(fetch(url))["hits"]
    hits_by_id = {hit["objectID"]: hit for hit in hits}
    threads = [hn_parse_search_hit(hits_by_id[tid]) for tid in thread_ids if tid in hits_by_id]

    return threads


def hn_fetch_threads(group: str = "news", page: int = 1) -> list[Message]:
    rex = re.compile(r'href="item\?id=(\d+)"')

    html = fetch(f"https://news.ycombinator.com/{group}?p={page}")
    thread_ids = list(dict.fromkeys(match.group(1) for match in rex.finditer(html)))

    return hn_fetch_threads_by_id(thread_ids)


def hn_fetch_new_threads(page: int = 1) -> list[Message]:
    url = f"https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=30&page={page}"
    hits = json.loads(fetch(url))["hits"]

    return [hn_parse_search_hit(hit) for hit in hits]


def hn_fetch_thread(entry_id: Union[str, int]) -> Message:
    resp = fetch(f"http://hn.algolia.com/api/v1/items/{entry_id}")
    entry: HNEntry = json.loads(resp)
    return hn_parse_entry(entry)


def lb_parse_thread(thread: LBThread) -> Message:
    comments = {}

    thread_body = f"{thread['description']}"
    thread_body = f"<p>{thread['url']}</p>{thread_body}" if thread["url"] else thread_body

    ret = Message(
        msg_id=f"{thread['short_id']}@lb",
        thread_id=f"{thread['short_id']}@lb",
        content_location=thread["short_id_url"],
        date=datetime.fromtimestamp(datetime.fromisoformat(thread["created_at"]).timestamp()),
        author=thread["submitter_user"],
        title=thread["title"],
        body=thread_body,
        children=None if thread.get("comments") is None else [],
        total_comments=thread["comment_count"] + 1,
    )

    for comment in thread.get("comments", []) or []:
        comments[comment["short_id"]] = Message(
            msg_id=f"{comment['short_id']}@lb",
            thread_id=f"{thread['short_id']}@lb",
            content_location=comment["url"],
            date=datetime.fromtimestamp(datetime.fromisoformat(comment["created_at"]).timestamp()),
            author=comment["commenting_user"],
            title=f"Re: {thread['title']}",
            body=comment["comment"],
            children=[],
        )

    for comment in thread.get("comments", []) or []:
        msg = comments[comment["short_id"]]
        parent_msg = comments[comment["parent_comment"]] if comment["parent_comment"] else ret
        msg.parent = parent_msg
        if parent_msg.children is not None:
            parent_msg.children.append(msg)

    return ret


def lb_fetch_threads(group: str = "", page: int = 1) -> list[Message]:
    group_path = f"{group}/" if group else ""
    resp = fetch(f"https://lobste.rs/{group_path}page/{page}.json")
    threads: list[LBThread] = json.loads(resp)

    return [lb_parse_thread(thread) for thread in threads]


def lb_fetch_thread(entry_id: str) -> Message:
    resp = fetch(f"https://lobste.rs/s/{entry_id}.json")
    thread: LBThread = json.loads(resp)

    return lb_parse_thread(thread)


def group_set_page(group: Group, page: int) -> Group:
    return dataclasses.replace(group, page=page)


def group_advance_page(group: Group, offset: int = 1) -> Group:
    return group_set_page(group, page=max(1, group.page + offset))


def group_fetch_starred_threads(db: DB, page: int = 1) -> list[Message]:
    thread_ids = db_load_starred_thread_ids(db, page)
    threads_by_provider_id: dict[str, list[str]] = {}
    threads = []

    for source_id, provider_id in (t.split("@") for t in thread_ids):
        threads_by_provider_id.setdefault(provider_id, list()).append(source_id)

    for provider_id, thread_ids in threads_by_provider_id.items():
        provider = PROVIDERS[provider_id]
        threads += provider.fetch_threads_by_id(thread_ids)

    threads.sort(key=lambda x: x.date, reverse=True)

    return threads


def group_fetch_thread(thread_id: str) -> Message:
    (msg_id, provider_id) = thread_id.split("@")
    provider = PROVIDERS[provider_id]

    return provider.fetch_thread(msg_id)


def group_for_msg_url(url: str) -> Group:
    if (match := HN_URL_REX.match(url)) is not None:
        msg_id = match[1]
        return Group(label="", fetch=lambda *x: hn_fetch_threads_by_id([msg_id]))

    msg = "Unknown URL, available patterns: \n" + "\n".join(f"- {r.pattern}" for r in [HN_URL_REX])
    raise ExitException(1, msg)


def app_safe_run(app: AppState, fn: Callable[[], T], flash: Optional[str]) -> Optional[T]:
    if flash is not None:
        app_show_flash(app, flash)

    ret = None

    try:
        ret = fn()
    except Exception as e:
        logging.debug("\n".join(traceback.format_exception(e)))
        app_show_flash(app, f"Error: {e}")
    else:
        if flash is not None:
            app_show_flash(app, None)

    return ret


def app_refresh_message(app: AppState) -> None:
    app.pager_offset = 0

    # Converting html to lines lazily on render for easier debugging
    if (msg := app.selected_message) is not None:
        msg.lines = msg_build_raw_lines(msg) if app.raw_mode else msg_build_lines(msg)


def app_select_message(app: AppState, message: Optional[Message], show_pager: bool = False) -> None:
    app.selected_message = message

    app_refresh_message(app)

    if message is None or message.body is None:
        app.pager_visible = False
        return

    if show_pager:
        app.pager_visible = True

    if app.pager_visible:
        message.flags.read = True
        db_save_message(app.db, message)
        db_load_read_comments(app.db, {message.thread_id: app.messages_by_id[message.thread_id]})


def app_load_messages(
    app: AppState, messages: list[Message], selected_message_id: Optional[str] = None, show_pager: bool = False
) -> None:
    if selected_message_id is None and app.selected_message is not None:
        selected_message_id = app.selected_message.msg_id

    selected_message = None

    for i, message in enumerate(messages):
        message.index_position = i

        if message.msg_id == selected_message_id:
            selected_message = message

    if selected_message is None and len(messages) > 0:
        selected_message = messages[0]

    app.messages = messages
    app.messages_by_id = {msg.msg_id: msg for msg in messages}

    db_load_message_flags(app.db, app.messages_by_id)
    db_load_read_comments(app.db, app.messages_by_id)

    app_select_message(app, selected_message, show_pager)


def app_load_group(app: AppState, group: Group) -> None:
    fn = partial(group.fetch, app.db, group.page)
    flash = f"Fetching stories from '{group.label}' (page {group.page})..."

    if (messages := app_safe_run(app, fn, flash=flash)) is None:
        return

    app_load_messages(app, messages)
    app.group = group


def app_close_thread(app: AppState) -> None:
    selected_thread_id = app.selected_message.thread_id if app.selected_message else None
    filtered_messages = [msg_unload(msg) for msg in app.messages if msg.is_thread]

    app_load_messages(app, filtered_messages, selected_message_id=selected_thread_id)


def app_open_thread(app: AppState, thread_message: Message) -> None:
    fn = partial(group_fetch_thread, thread_message.thread_id)
    flash = f"Fetching thread '{thread_message.thread_id}'..."

    if (new_thread_message := app_safe_run(app, fn, flash=flash)) is None:
        return

    app_close_thread(app)

    index_pos = thread_message.index_position
    thread_messages = list(msg_flatten_thread(new_thread_message, ascii=app.ascii))
    new_thread_message.total_comments = len(thread_messages)
    messages = app.messages[:index_pos] + thread_messages + app.messages[index_pos + 1 :]  # noqa: E203

    app_load_messages(app, messages, selected_message_id=thread_message.msg_id, show_pager=True)


def app_update_layout(app: AppState) -> None:
    lt = app.layout

    (lt.lines, lt.cols) = app.screen.getmaxyx()

    if lt.lines < 12 or lt.cols < 80:
        raise ExitException(1, "At least 80x12 terminal is required")

    max_index_height = lt.lines - 3
    lt.index_height = (max_index_height // 3) if app.pager_visible else max_index_height

    lt.middle_menu_row = lt.index_start + lt.index_height if app.pager_visible else None
    lt.pager_start = lt.index_start + lt.index_height + 1 if app.pager_visible else None
    lt.pager_height = lt.lines - lt.pager_start - 2 if lt.pager_start is not None else None

    lt.bottom_menu_row = lt.lines - 2
    lt.flash_menu_row = lt.lines - 1


def app_show_help_screen(app: AppState) -> None:
    max_lines = app.layout.lines - 4

    help_lines = HELP_SCREEN.split("\n")
    help_pages = ["\n".join(lines) for lines in list_chunk(help_lines, max_lines)]

    for page in help_pages:
        app.screen.erase()
        app.screen.addstr(0, 0, "Available commands:\n\n")
        app.screen.addstr(page)
        app.screen.addstr("\n\nPress any key to continue...")
        app.screen.refresh()
        app.screen.getch()


def app_show_links_screen(app: AppState) -> None:
    lines = app.selected_message.lines if app.selected_message is not None else []

    # Max amount of keys is 21 to fit on 25-line terminals
    keys = "1234567890abcdefghijk"
    urls = URL_REX.findall(" ".join(lines))
    items = dict(zip((ord(k) for k in keys), urls))

    if len(items) == 0:
        return app_show_flash(app, "No links available for opening")

    app.screen.erase()
    app.screen.addstr(0, 0, "Select link to open:")

    for i, (key, url) in enumerate(items.items()):
        app.screen.addstr(i + 2, 0, f"{chr(key)} - {url}")

    app.screen.addstr(i + 4, 0, "To change browser run: BROWSER='firefox %s' ./retronews.py")
    app.screen.refresh()

    key = app.screen.getch()

    if key not in items.keys():
        return app_show_flash(app, "Unknown key")

    url = items[key]

    app_show_flash(app, f"Opening {url}")
    webbrowser.open(url)

    # Refresh window in case a terminal browser was used
    app.screen.clearok(True)


def app_show_flash(app: AppState, flash: Optional[str]) -> None:
    app.flash = flash
    app_render(app)


def app_prompt(app: AppState, prompt: str) -> str:
    lt = app.layout

    app.screen.insstr(lt.flash_menu_row, 0, prompt.ljust(lt.cols))
    app.screen.refresh()

    curses.curs_set(1)
    win = curses.newwin(1, lt.cols - len(prompt), lt.flash_menu_row, len(prompt))

    textbox = curses.textpad.Textbox(win)
    textbox.stripspaces = True
    ret = textbox.edit().strip()

    del win
    curses.curs_set(0)

    return ret


def app_render_index_row(app: AppState, row: int, message: Message) -> None:
    cols = app.layout.cols
    date = message.date.strftime("%Y-%m-%d %H:%M")
    author = (message.author or "<unknown>")[:10].ljust(10)

    is_response = message.title.startswith("Re:") and not message.is_thread
    is_selected = message == app.selected_message
    hide_title = is_response and row > app.layout.index_start and not message.flags.starred and not is_selected
    title = "" if hide_title else text_clean(message.title, ascii=app.ascii)

    unread = (
        str(max(min(message.total_comments - message.read_comments, 9999), 0)).rjust(4) if message.is_thread else "    "
    )

    app.screen.insstr(row, 0, f"[{date}]  [{author}]  [{unread}]  {message.index_tree}{title}")

    if is_selected:
        cursor_attr = curses.A_REVERSE if app.monochrome else 0
        app.screen.chgat(row, 0, cols, app.colors["cursor"] | cursor_attr)
    else:
        read_attr = 0 if message.is_shown_as_read else curses.A_BOLD
        subject_attr = app.colors["starred_subject"] if message.flags.starred else app.colors["default"]
        subject_attr = subject_attr | read_attr

        app.screen.chgat(row, 1, 16, app.colors["date"] | read_attr)
        app.screen.chgat(row, 21, 10, app.colors["author"] | read_attr)
        app.screen.chgat(row, 35, 4, app.colors["unread_comments"] | read_attr)
        app.screen.chgat(row, 42, len(message.index_tree), app.colors["tree"])
        app.screen.chgat(row, 42 + len(message.index_tree), cols - 42 - len(message.index_tree), subject_attr)


def app_render_index(app: AppState) -> None:
    height = app.layout.index_height

    offset = app.selected_message.index_position - height // 2 if app.selected_message else 0
    offset = min(offset, len(app.messages) - height)
    offset = max(offset, 0)

    rows_to_render = min(height, len(app.messages) - offset)

    for i in range(rows_to_render):
        app_render_index_row(app, app.layout.index_start + i, app.messages[i + offset])


def app_get_pager_line_attr(app: AppState, line: str) -> int:
    if line.startswith("Content-Location: "):
        return app.colors["tree"]
    elif line.startswith("Date: "):
        return app.colors["date"]
    elif line.startswith("From: "):
        return app.colors["author"]
    elif line.startswith("Subject: "):
        return app.colors["header_subject"]
    elif line.startswith(">>") or line.startswith("> >"):
        return app.colors["nested_quote"]
    elif line.startswith(">"):
        return app.colors["quote"]
    elif line.startswith("| "):
        return app.colors["code"]
    elif line == "~":
        return app.colors["empty_pager_line"]
    elif line == "<deleted>" or line == "[dead]":
        return app.colors["deleted_message_pager_line"]
    else:
        return 0


def app_render_pager_line(app: AppState, row: int, line: str) -> None:
    hl_lines = not app.raw_mode
    line_attr = app_get_pager_line_attr(app, line) if hl_lines else 0
    hl_urls = line_attr == 0 and hl_lines

    line = text_clean(line, ascii=app.ascii)

    app.screen.move(row, 0)
    app.screen.clrtoeol()
    app.screen.move(row, 0)

    for part in text_split_urls(line):
        is_url = URL_REX.fullmatch(part)
        part_attr = app.colors["url"] if is_url and hl_urls else line_attr
        app.screen.addstr(part, part_attr)


def app_render_pager(app: AppState) -> None:
    message = app.selected_message
    start = app.layout.pager_start
    height = app.layout.pager_height

    if message is None or start is None or height is None:
        return

    for i in range(height):
        line = list_get(message.lines, i + app.pager_offset, "~")
        app_render_pager_line(app, i + start, line)


def app_render_top_menu(app: AppState) -> None:
    lt = app.layout
    cols = lt.cols
    base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD
    app.screen.insstr(lt.top_menu_row, 0, HELP_MENU[:cols].ljust(cols), app.colors["menu"] | base_attr)


def app_render_middle_menu(app: AppState) -> None:
    if (row := app.layout.middle_menu_row) is None:
        return

    if (message := app.selected_message) is None:
        return

    if (thread_message := app.messages_by_id.get(message.thread_id)) is None:
        return

    cols = app.layout.cols
    total = thread_message.total_comments
    unread = total - thread_message.read_comments

    text = f"--({unread}/{total} unread)"
    if thread_message.flags.starred:
        text += "--(starred thread)"
    if app.raw_mode:
        text += "--(raw mode on)"
    text = text[:cols].ljust(cols, "-")

    base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD
    app.screen.insstr(row, 0, text, app.colors["menu"] | base_attr)


def app_render_bottom_menu(app: AppState) -> None:
    lt = app.layout
    base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD

    app.screen.chgat(lt.bottom_menu_row, 0, lt.cols, app.colors["menu"] | base_attr)
    app.screen.move(lt.bottom_menu_row, 0)

    for i, group in enumerate(GROUP_TABS):
        is_active = group.label == app.group.label
        item_attr = app.colors["menu_active"] | curses.A_BOLD if is_active else app.colors["menu"]
        item_attr = item_attr | base_attr
        app.screen.addstr(f"{i+1}:{group.label}", item_attr)
        app.screen.addstr("  ", app.colors["menu"] | base_attr)

    page_text = f"page: {app.group.page}"
    app.screen.insstr(lt.bottom_menu_row, lt.cols - len(page_text), page_text, app.colors["menu"] | base_attr)


def app_render(app: AppState) -> None:
    app_update_layout(app)
    app.screen.erase()
    app_render_index(app)
    app_render_pager(app)
    app_render_top_menu(app)
    app_render_middle_menu(app)
    app_render_bottom_menu(app)
    app.screen.insstr(app.layout.flash_menu_row, 0, app.flash or "")
    app.screen.refresh()


def app_init_colors(app: AppState) -> None:
    if app.monochrome:
        app.colors = defaultdict(lambda: 0)
        return

    try:
        curses.use_default_colors()
        for i, (name, (fg, bg)) in enumerate(COLORS.items()):
            curses.init_pair(i + 1, fg, bg)
            app.colors[name] = curses.color_pair(i + 1)
    except curses.error:
        app.colors = defaultdict(lambda: 0)
        app.monochrome = True


def app_main(screen: Window, db: DB, group: Group, ascii: bool, monochrome: bool) -> int:
    curses.curs_set(0)

    app = AppState(screen=screen, db=db, group=group, ascii=ascii, monochrome=monochrome)

    app_init_colors(app)
    app_load_group(app, app.group)

    while True:
        app_render(app)
        app.flash = ""
        c = app.screen.getch()
        KEY_BINDINGS.get(c, cmd_unknown)(app)


def setup_logging(path: Optional[str]) -> None:
    if path is None:
        return logging.disable()

    format = "%(asctime)s %(levelname)s: %(message)s"
    stream = sys.stderr if path == "-" else open(path, "a")
    logging.basicConfig(format=format, level="DEBUG", stream=stream)
    logging.debug("Session started")


def run_rcfile(path: str) -> None:
    path = os.path.expanduser(path)

    if not os.path.isfile(path):
        return

    code = compile(open(path).read(), path, "exec")
    exec(code, {"retronews": sys.modules[__name__]})


if __name__ == "__main__":
    tab_choices = range(1, len(GROUP_TABS) + 1)

    ap = argparse.ArgumentParser(
        formatter_class=lambda prog: argparse.ArgumentDefaultsHelpFormatter(prog, max_help_position=32)
    )
    ap.add_argument("--ascii", action="store_true", help="show only ascii characters")
    ap.add_argument("--monochrome", action="store_true", help="disable colors")
    ap.add_argument("-c", "--rcfile", metavar="PATH", default="~/.retronewsrc.py", help="optional startup code path")
    ap.add_argument("-d", "--db", metavar="PATH", default="~/.retronews.db", help="database path")
    ap.add_argument("-l", "--logfile", metavar="PATH", default=None, help="debug logfile path")
    ap.add_argument("-t", "--tab", metavar="TAB", type=int, default=1, choices=tab_choices, help="initial tab")
    ap.add_argument("-r", "--render", metavar="PATH", default=None, help="render raw html message and quit")
    ap.add_argument("-m", "--msg", metavar="URL", default=None, help="render message from URL")
    args = ap.parse_args()

    setup_logging(args.logfile)
    run_rcfile(args.rcfile)

    if (path := args.render) is not None:
        with open(path) as fp:
            print(html_render(fp.read()))
        sys.exit(0)

    try:
        db = db_init(args.db)

        if args.msg:
            group = group_for_msg_url(args.msg)
        else:
            group = GROUP_TABS[args.tab - 1]

        ascii = args.ascii
        monochrome = args.monochrome or "NO_COLOR" in os.environ

        ret = curses.wrapper(app_main, db=db, group=group, ascii=ascii, monochrome=monochrome)
    except ExitException as e:
        if e.message:
            sys.stderr.write(e.message + "\n")
        ret = e.code
    except BaseException as e:
        sys.stderr.write("\n".join(traceback.format_exception(e)))
        ret = 1
    finally:
        db.close()
        sys.exit(ret)


================================================
FILE: tests/test_hn_01.html
================================================
<p>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p><p>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p>

================================================
FILE: tests/test_hn_01.out
================================================
Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

================================================
FILE: tests/test_hn_02.html
================================================
<p>&gt;&gt;Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p><p>&gt; Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p><p>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p>


================================================
FILE: tests/test_hn_02.out
================================================
>>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
>>te. Noster nominati recteque no has.

> Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
> te. Noster nominati recteque no has.

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

================================================
FILE: tests/test_hn_03.html
================================================
<a href="https://example.com/foo/bar">https://example.com/foo...</a>


================================================
FILE: tests/test_hn_03.out
================================================
https://example.com/foo/bar

================================================
FILE: tests/test_hn_04.html
================================================
<p>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus</p><p>[0]: <a>https://long.long.long.long.long.long.long.long.long.long.long.example.com</a></p><p>[1] - <a>https://long.long.long.long.long.long.long.long.long.long.long.example.com</a></p><p>[2] <a>https://long.long.long.long.long.long.long.long.long.long.long.example.com</a></p>


================================================
FILE: tests/test_hn_04.out
================================================
Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus

[0]: https://long.long.long.long.long.long.long.long.long.long.long.example.com

[1] - https://long.long.long.long.long.long.long.long.long.long.long.example.com

[2] https://long.long.long.long.long.long.long.long.long.long.long.example.com

================================================
FILE: tests/test_hn_05.html
================================================
<p>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p><p><pre><code>    def hello():
        print('Hello World')
        return None
</code></pre>
Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p><p><pre><code>    def hello():
        print('Hello World')
        return None
</code></pre>
Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.</p>


================================================
FILE: tests/test_hn_05.out
================================================
Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

| def hello():
|     print('Hello World')
|     return None

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

| def hello():
|     print('Hello World')
|     return None

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus
te. Noster nominati recteque no has.

================================================
FILE: tests/test_hn_06.html
================================================
<p>Lorem ipsum dolor sit amet, pro eu soleat civibus.</p><p><pre><code>    lambda L: [] if L==[] else qsort([x for x in L[1:] if x< L[0]]) + L[0:1] + qsort([x for x in L[1:] if x>=L[0]])
</code></pre>Mel quas sensibus te. Noster nominati recteque no has.</p>


================================================
FILE: tests/test_hn_06.out
================================================
Lorem ipsum dolor sit amet, pro eu soleat civibus.

| lambda L: [] if L==[] else qsort([x for x in L[1:] if x< L[0]]) + L[0:1] + qsort([x for x in L[1:] if x>=L[0]])

Mel quas sensibus te. Noster nominati recteque no has.

================================================
FILE: tests/test_lb_01.html
================================================
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>


================================================
FILE: tests/test_lb_01.out
================================================
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

================================================
FILE: tests/test_lb_02.html
================================================
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem <code>ipsum</code> dolor sit amet, <code>consectetur adipiscing</code> elit, sed do eiusmod tempor</p>
<pre><code>def hello_world():
  print('hello')
  if True:
    print('world')
  print('!!!')

hello_world()
</code></pre>


================================================
FILE: tests/test_lb_02.out
================================================
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
> adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem `ipsum` dolor sit amet,
`consectetur adipiscing` elit, sed do eiusmod tempor

| def hello_world():
|   print('hello')
|   if True:
|     print('world')
|   print('!!!')
| 
| hello_world()

================================================
FILE: tests/test_lb_03.html
================================================
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<ul>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.</li>
<li>Lorem:
<ul>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.</li>
<li>Lorem ipsum dolor sit amet, consectetur.</li>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.</li>
</ul>
</li>
</ul>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>


================================================
FILE: tests/test_lb_03.out
================================================
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

- Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
  eiusmod tempor.
- Lorem:

  - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
    eiusmod tempor.
  - Lorem ipsum dolor sit amet, consectetur.
  - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
    eiusmod tempor.

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

================================================
FILE: tests/test_lb_04.html
================================================
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, <a href="https://example.com/" rel="ugc">example</a> sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur <code>adipiscing elit</code>, sed do eiusmod tempor</p>
<blockquote>
<p>
  Lorem ipsum <em>dolor</em> <b>sit<br>dolor</b> amet, consectetur

   adipiscing  
  
  
    elit
    </p>
<hr>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>

================================================
FILE: tests/test_lb_04.out
================================================
Lorem ipsum dolor sit amet, consectetur adipiscing elit

> Lorem ipsum dolor sit amet, consectetur adipiscing elit, example
> https://example.com/ sed do eiusmod tempor. Lorem ipsum dolor sit
> amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem
> ipsum dolor sit amet, consectetur `adipiscing elit`, sed do eiusmod
> tempor
> 
>> Lorem ipsum /dolor/ *sit
>> dolor* amet, consectetur adipiscing elit
>> 
>> ------------------------------------------------------------------
>> 
>> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
>> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
>> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet,
>> consectetur adipiscing elit, sed do eiusmod tempor
> 
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
> adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit

================================================
FILE: tests/test_lb_05.html
================================================
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<hr>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<hr>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>


================================================
FILE: tests/test_lb_05.out
================================================
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

----------------------------------------------------------------------

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

----------------------------------------------------------------------

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

================================================
FILE: tests/test_lb_06.html
================================================
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<blockquote>
<ul>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit</li>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit</li>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit</li>
</ul>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>


================================================
FILE: tests/test_lb_06.out
================================================
> Lorem ipsum dolor sit amet, consectetur adipiscing elit

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
> adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit

> - Lorem ipsum dolor sit amet, consectetur adipiscing elit
> - Lorem ipsum dolor sit amet, consectetur adipiscing elit
> - Lorem ipsum dolor sit amet, consectetur adipiscing elit

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

================================================
FILE: tests/test_lb_07.html
================================================
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<ol>
<li>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</li>
<li>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</li>
<li>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</li>
</ol>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>


================================================
FILE: tests/test_lb_07.out
================================================
Lorem ipsum dolor sit amet, consectetur adipiscing elit

- Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
  eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
  elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor
- Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
  eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
  elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor
- Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
  eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
  elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
  adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
adipiscing elit, sed do eiusmod tempor

================================================
FILE: tests/test_lb_08.html
================================================
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<blockquote>
<blockquote>
<ol start="3">
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit</li>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</li>
<li>Lorem ipsum dolor sit amet, consectetur adipiscing elit</li>
</ol>
</blockquote>
</blockquote>
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor</p>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>


================================================
FILE: tests/test_lb_08.out
================================================
Lorem ipsum dolor sit amet, consectetur adipiscing elit

>> - Lorem ipsum dolor sit amet, consectetur adipiscing elit
>> - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
>>   eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
>>   adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit
>>   amet, consectetur adipiscing elit, sed do eiusmod tempor
>> - Lorem ipsum dolor sit amet, consectetur adipiscing elit

> Lorem ipsum dolor sit amet, consectetur adipiscing elit
> 
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing
> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur
> adipiscing elit, sed do eiusmod tempor

Lorem ipsum dolor sit amet, consectetur adipiscing elit

================================================
FILE: tests/test_lb_09.html
================================================
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
</blockquote>
<blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<pre><code>def hello_world():
  print('hello')
  if True:
    print('world')
  print('!!!')

hello_world()
</code></pre>
</blockquote>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>
<pre><code>def hello_world():
  print('hello')
  if True:
    print('world')
  print('!!!')

hello_world()
</code></pre>
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit</p>


================================================
FILE: tests/test_lb_09.out
================================================
> Lorem ipsum dolor sit amet, consectetur adipiscing elit

> Lorem ipsum dolor sit amet, consectetur adipiscing elit
> 
> | def hello_world():
> |   print('hello')
> |   if True:
> |     print('world')
> |   print('!!!')
> | 
> | hello_world()

Lorem ipsum dolor sit amet, consectetur adipiscing elit

| def hello_world():
|   print('hello')
|   if True:
|     print('world')
|   print('!!!')
| 
| hello_world()

Lorem ipsum dolor sit amet, consectetur adipiscing elit

================================================
FILE: tests.py
================================================
import os
import subprocess
import unittest

import retronews

TC_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "tests")


class TestHtmlRender(unittest.TestCase):
    maxDiff = None

    def checkRendering(self, name: str):
        html_path = os.path.join(TC_DIR, f"{name}.html")
        out_path = os.path.join(TC_DIR, f"{name}.out")

        with open(html_path) as fp:
            html = fp.read()

        actual = retronews.html_render(html)

        if not os.path.exists(out_path):
            with open(out_path, "w") as fp:
                fp.write(actual)
            return

        cmd = ["diff", "-Nru", "--color=always", out_path, "-"]
        proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
        stdout, stderr = proc.communicate(input=actual)

        if proc.returncode != 0:
            self.fail(f"Unexpected rendering output\n{stdout}")


def setup_test_cases():
    tcs = [x.split(".")[0] for x in sorted(os.listdir(TC_DIR)) if x.endswith(".html")]

    for tc in tcs:
        setattr(TestHtmlRender, tc, lambda self, tc=tc: self.checkRendering(tc))


if __name__ == "__main__":
    setup_test_cases()
    unittest.main()