leafee98/static_deployer
1
1
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Compare commits

base: leafee98:main
Branches Tags
leafee98:main
leafee98:v0.0
..
compare: leafee98:v0.0
Branches Tags
leafee98:main
leafee98:v0.0

No commits in common. "main" and "v0.0" have entirely different histories.
main ... v0.0

4 changed files with 0 additions and 407 deletions
Show stats Expand all files Collapse all files

106
README.md
View file

@ -1,106 +0,0 @@
# Static Deplyer
A simple daemon help with extracting tar files to specific place
and create a symbolic link to it, the tar files should be transferred
to this daemon by http.
## Caution
No `data-form` or `x-www-urlencoded-form` supported, so the tar must be transferred
just in POST body, ie: `curl --data-binary @file.tar.gz`.
No authorization implemented, so a reverse proxy with auth is recommanded.
Tar file with sub directories with be extracted as is, so do not contain parent
directory when you create tar.
## Usage
```
$ ./main.py --help
usage: main.py [-h] --archive-dir ARCHIVE_DIR --extract-dir EXTRACT_DIR --symlink-path
SYMLINK_PATH [--keep-extract KEEP_EXTRACT] [--keep-archive KEEP_ARCHIVE]
--port PORT [--temp-dir TEMP_DIR]
options:
-h, --help show this help message and exit
--archive-dir ARCHIVE_DIR
directory to save archives
--extract-dir EXTRACT_DIR
directory to save extracted files
--symlink-path SYMLINK_PATH
path of symlink which redirect to extracted archive
--keep-extract KEEP_EXTRACT
Number of extracted archives to keep, 0 mean never vacuum
--keep-archive KEEP_ARCHIVE
Number of archives to keep, 0 mean never vacuum
--port PORT listen port on 127.0.0.1, no authorization implemented so only
listen on 127.0.0.1 for safety
--temp-dir TEMP_DIR path to save in-delivery archive
```
## Example
First start the daemon.
```
$ ./main.py --port 8080 --archive-dir archive --extract-dir extracted --symlink-path serve
INFO:root:Listening on 127.0.0.1:8080
INFO:root:Archive saves under: archive
INFO:root:Extract tar under: extracted
INFO:root:Keep 8 archives at most
INFO:root:Keep 4 extracted at most
INFO:root:Symbolic link location: serve
INFO:root:Temperory directory: /tmp
INFO:root:Starting httpd...
```
Then create a tar and upload it to this daemon.
**Note**: the tar shouldn't contain its parent directory, but `.` as parent
is acceptable. Or you can follow [this step](https://stackoverflow.com/a/39530409)
to create a more elegant tar.
```
$ mkdir tmp
$ cd tmp
$ echo 'Hello, world!' > index.html
$ tar --gzip -cf ../tmp.tar.gz .
$ tar -tf ../tmp.tar.gz
./
./index.html
$ curl --data-binary @../tmp.tar.gz http://localhost:8080/
Success
```
And the server side shows
```
...
INFO:root:Starting httpd...
INFO:FileManager:Temporarily save to /tmp/archive_2023-04-04T10:35:49.tar.gz
INFO:FileManager:Moving saved archive to archive/archive_2023-04-04T10:35:49.tar.gz
INFO:FileManager:Extracting to extracted/archive_2023-04-04T10:35:49
INFO:FileManager:Recreating symlink point to extracted/archive_2023-04-04T10:35:49
INFO:FileManager:Vacuuming archive, keep the 8 lastest
INFO:FileManager:Vacuuming extract, keep the 4 lastest
INFO:FileManager:Deploy success
127.0.0.1 - - [04/Apr/2023 10:35:49] "POST / HTTP/1.1" 200 -
```
Finally the directory looks like (omit unrelated directories):
```
.
├── archive
│ └── archive_2023-04-04T10:35:49.tar.gz
├── extracted
│ └── archive_2023-04-04T10:35:49
│ └── index.html
└── serve -> extracted/archive_2023-04-04T10:35:49
```
## Use Case
When you hold a static site and want to update its content easily,
like just uploading a tar and automatically deployed.

272
main.py
View file

@ -1,272 +0,0 @@
#!/usr/bin/env python3
"""
This Listen a port on 127.0.0.1 or [::1], (so no authorization
implement required), receive a .tar.gz file and extract it to
a specific path, and make it a soft link to the extracted files.
Use the following command to upload the file content:
curl -H "Content-Type: application/octet-stream" --data-binary @main.py http://localhost:8080
The file content must be a legal .tar.gz file. There must not be
a subdirectory to contain other files
"""
import os
import tarfile
import shutil
from pathlib import Path
from typing import BinaryIO, Union
from http.server import BaseHTTPRequestHandler, HTTPServer
from datetime import datetime
import argparse
import logging
class NotSymlinkException(Exception):
pass
class NotDirectoryException(Exception):
pass
class FileManager:
logger = logging.getLogger("FileManager")
archive_ext = ".tar.gz"
def __init__(self, archive_dir: str, extract_dir: str, symlink_path: str,
keep_archive: int, keep_extract: int, temp_dir: str = "/tmp"):
self.archive_dir = archive_dir
self.extract_dir = extract_dir
self.symlink_path = symlink_path
self.keep_archive = keep_archive
self.keep_extract = keep_extract
self.temp_dir = temp_dir
self._check_dirs()
def _check_dirs(self):
def check_dir(path):
p = Path(path)
if p.exists():
if not p.is_dir():
raise NotDirectoryException("{} exists and is not a directory".format(path))
else:
os.makedirs(path)
def check_symlink(path):
p = Path(path)
if p.exists():
if not p.is_symlink():
raise NotSymlinkException("{} exists and is not a symlink".format(path))
else:
check_dir(os.path.dirname(path))
check_dir(self.archive_dir)
check_dir(self.extract_dir)
check_dir(self.temp_dir)
check_symlink(self.symlink_path)
def _get_archive_name(self) -> str:
time_str = datetime.now().isoformat(timespec="seconds")
return f"archive_{time_str}{self.archive_ext}"
def _get_basename(self, filename: str) -> str:
if filename.endswith(self.archive_ext):
return filename[:-len(self.archive_ext)]
return filename
def _extract(self, archive_path: str, target_path: str) -> bool:
try:
with tarfile.open(archive_path, mode="r:gz") as tf:
tf.extractall(target_path)
except Exception as e:
self.logger.error("Failed to extract tar file: {}".format(e))
return False
return True
def save_file(self, src: BinaryIO, content_length: int) -> Union[str, None]:
archive_name = self._get_archive_name()
tgt_file = os.path.join(self.temp_dir, archive_name)
self.logger.info("Temporarily save to {}".format(tgt_file))
try:
f = open(tgt_file, "bw")
redirect_stream(src, f, content_length)
f.close()
except:
os.remove(tgt_file)
return None
final_file = os.path.join(self.archive_dir, archive_name)
self.logger.info("Moving saved archive to {}".format(final_file))
shutil.move(tgt_file, final_file)
return final_file
def deploy(self, archive_path: str) -> bool:
extract_dir = os.path.join(self.extract_dir,
self._get_basename(os.path.basename(archive_path)))
self.logger.info("Extracting to {}".format(extract_dir))
os.mkdir(extract_dir)
if not self._extract(archive_path, extract_dir):
self.logger.error("Failed to extract archive {} to {}"
.format(archive_path, extract_dir))
return False
self.logger.info("Recreating symlink point to {}".format(extract_dir))
if Path(self.symlink_path).exists():
os.remove(self.symlink_path)
os.symlink(extract_dir, self.symlink_path)
return True
def _vacuum_single(self, dirname: str, keep_count: int, rm_dir: bool) -> None:
files = os.listdir(dirname)
files.sort()
for f in files[:-keep_count]:
full_path = os.path.join(dirname, f)
self.logger.info("Removing {}".format(full_path))
if rm_dir:
shutil.rmtree(full_path)
else:
os.remove(full_path)
def vacuum(self) -> None:
if self.keep_archive > 0:
self.logger.info("Vacuuming archive, keep the {} lastest".format(self.keep_archive))
self._vacuum_single(self.archive_dir, self.keep_archive, False)
if self.keep_extract > 0:
self.logger.info("Vacuuming extract, keep the {} lastest".format(self.keep_extract))
self._vacuum_single(self.extract_dir, self.keep_extract, True)
def handle(self, instream: BinaryIO, content_length: int) -> bool:
archive_path = self.save_file(instream, content_length)
if archive_path is None:
self.logger.error("Failed to save file. Aborted!")
return False
if not self.deploy(archive_path):
self.logger.error("Failed to extract or create symlink. Aborted!")
return False
self.vacuum()
self.logger.info("Deploy success")
return True
global_mgr: FileManager
def redirect_stream(src: BinaryIO, tgt: BinaryIO, size: int) -> None:
block_size = 4 * 1024 * 1024 # 4MB
cache = src.read(size % block_size)
tgt.write(cache)
size -= size % block_size
while size > 0:
cache = src.read(block_size)
tgt.write(cache)
size -= block_size
class S(BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
logger = logging.getLogger("HttpHandler")
def __init__(self, *args, **kwargs):
super(S, self).__init__(*args, **kwargs)
def _set_response(self):
self.send_response(200)
self.send_header("Content-Type", "text/plaintext")
self.end_headers()
def do_GET(self):
self.logger.info("Received GET request, Path: %s", str(self.path))
content = "Non-implemented".encode("utf-8")
self._write_response(403, "text/plaintext", content)
def _write_response(self, status_code: int, content_type: str, content: bytes):
self.send_response(status_code)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(content)))
self.end_headers()
self.wfile.write(content)
def do_POST(self):
content_length = int(self.headers["Content-Length"])
if global_mgr.handle(self.rfile, content_length):
content = "Success".encode("utf-8")
self._write_response(200, "text/plaintext", content)
else:
content = "Failed".encode("utf-8")
self._write_response(200, "text/plaintext", content)
self.wfile.flush()
def run(archive_dir: str, extract_dir: str, symlink_path: str,
keep_archive: int, keep_extract: int,
port: int = 8080, temp_dir: str = "/tmp"):
logging.basicConfig(level=logging.DEBUG)
address = "127.0.0.1"
logging.info("Listening on {}:{}".format(address, port))
logging.info("Archive saves under: {}".format(archive_dir))
logging.info("Extract tar under: {}".format(extract_dir))
logging.info("Keep {} archives at most".format(keep_archive))
logging.info("Keep {} extracted at most".format(keep_extract))
logging.info("Symbolic link location: {}".format(symlink_path))
logging.info("Temperory directory: {}".format(temp_dir))
global global_mgr
global_mgr = FileManager(archive_dir=archive_dir, extract_dir=extract_dir,
symlink_path=symlink_path, temp_dir=temp_dir,
keep_archive=keep_archive, keep_extract=keep_extract)
httpd = HTTPServer((address, port), S)
logging.info("Starting httpd...")
try:
httpd.serve_forever()
except KeyboardInterrupt:
pass
httpd.server_close()
logging.info("Stopping httpd...")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--archive-dir", dest="archive_dir", type=str,
required=True, help="directory to save archives")
ap.add_argument("--extract-dir", dest="extract_dir", type=str,
required=True, help="directory to save extracted files")
ap.add_argument("--symlink-path", dest="symlink_path", type=str,
required=True, help="path of symlink which redirect to extracted archive")
ap.add_argument("--keep-extract", dest="keep_extract", type=int,
default=4, help="Number of extracted archives to keep, 0 mean never vacuum")
ap.add_argument("--keep-archive", dest="keep_archive", type=int,
default=8, help="Number of archives to keep, 0 mean never vacuum")
ap.add_argument("--port", dest="port", type=int,
required=True, help="listen port on 127.0.0.1, " +
"no authorization implemented so only listen on 127.0.0.1 for safety")
ap.add_argument("--temp-dir", dest="temp_dir", type=str,
default="/tmp", help="path to save in-delivery archive")
args = ap.parse_args()
run(archive_dir=args.archive_dir,
extract_dir=args.extract_dir,
symlink_path=args.symlink_path,
keep_archive=args.keep_archive,
keep_extract=args.keep_extract,
temp_dir=args.temp_dir,
port=args.port)

11
scripts/create_archive.sh
View file

@ -1,11 +0,0 @@
#!/usr/bin/env bash
# Create an archive named with "$2", containing all files under "$1"
# and no subdir containing all others
if (( $# != 2 )) ; then
echo "parameters not meet"
exit 1
fi
find "$1" -mindepth 1 -not -type d -printf "%P\n" | tar -C "$1" -cf "$2" --gzip -T -

18
scripts/deploy.sh
View file

@ -1,18 +0,0 @@
#!/usr/bin/env bash
# The http server not support data-form or x-www-url-encoded-form,
# so the archive file must just in the post body, so use --data-binary
# as curl argument.
#
# $1 is the archive path to be sent to http server
# $2 is the http server (including schema, host, port and path)
#
# Example:
# curl --data-binary @public.tar.gz http://localhost:8080/
if (( $# != 2 )) ; then
echo "parameter not meet"
exit 1
fi
curl --data-binary @"$1" "$2"