Beiran

Data Models

Module of Beirand data models. Beirand data models use Peewee ORM.

class beiran.models.base.BaseModel(*args, **kwargs)[source]

Bases: peewee.Model

Base model object having common attributes, to be extended by data models.

DoesNotExist

alias of BaseModelDoesNotExist

Deserialize model from python dict

param _dict:python dict represents obj
type _dict:dict
returns:model object
rtype:BaseModel
id = <peewee.AutoField object>

Serialize model to python dict

param **kwargs:model attributes
returns:serialized model object
rtype:(dict)

Update model object with given obj

param obj:new object
type obj:BaseModel
class beiran.models.base.JSONStringField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, db_column=None, _hidden=False)[source]

Bases: peewee.TextField

A basic JSON Field based on text field

dict to string

string to python dict

Module for Node Data Model

class beiran.models.node.Node(*args, **kwargs)[source]

Bases: beiran.models.base.BaseModel

Node is a member of Beiran Cluster

DoesNotExist

alias of NodeDoesNotExist

STATUS_CLOSING = 'closing'
STATUS_CONNECTING = 'connecting'
STATUS_INIT = 'init'
STATUS_LOST = 'lost'
STATUS_NEW = 'new'
STATUS_OFFLINE = 'offline'
STATUS_ONLINE = 'online'
STATUS_READY = 'ready'
STATUS_SYNCING = 'syncing'
STATUS_UNKNOWN = 'unknown'

Initialize self. See help(type(self)) for accurate signature.

Update with or create a new node object from provided node object. :param node: node object :type node: Node

returns:node object
rtype:(Node)
address

Returns peer address of node in beiran address format.

If force is True, it gets from db.

Parameters:force (bool) – from db or not
Returns:peer address string
Return type:(str)
architecture = <peewee.CharField object>

Query for all node connections ordered by last seen time :returns: query object.

Deserialize model from python dict

param _dict:python dict represents obj
type _dict:dict
returns:model object
rtype:BaseModel

Get a list of connection details of node ordered by last seen time :returns: (list) list of PeerAddress of node

Get the latest PeerAddress of node :returns: (PeerAddress) connection object

hostname = <peewee.CharField object>
ip_address = <peewee.CharField object>
ip_address_6 = <peewee.CharField object>
last_sync_version = <peewee.IntegerField object>
os_type = <peewee.CharField object>
os_version = <peewee.CharField object>
port = <peewee.IntegerField object>

Set or get address of node

param address:beiran address string
type address:str
returns:beiran address string
rtype:(str)
status = <peewee.CharField object>

Serialize model to python dict

param **kwargs:model attributes
returns:serialized model object
rtype:(dict)
url

Generates node advertise url using ip_address, port and uuid properties

url_without_uuid

Generates node advertise url using ip_address, port properties

uuid = <peewee.UUIDField object>
version = <peewee.CharField object>
class beiran.models.node.PeerAddress(*args, **kwargs)[source]

Bases: beiran.models.base.BaseModel

Data model for connection details of Nodes

DoesNotExist

alias of PeerAddressDoesNotExist

Initialize self. See help(type(self)) for accurate signature.

Update with or create a new peer_address object from provided information.

param uuid:uuid
type uuid:str
param address:address
type address:str
param discovery:
 discovery channel
type discovery:str
param paconfig:paconfig dict
type paconfig:dict

Returns:

address = <peewee.CharField object>

Build a node address with given host, port, protocol and uuid

param host:hostname
type host:str
param uuid:uuid of node
type uuid:str
param path:address path
type path:str
param port:service port
type port:int
param protocol:protocol, default http
type protocol:str
param socket:is unix socket? default False
type socket:str
returns:formatted address
rtype:str
config = <beiran.models.base.JSONStringField object>
discovery_method = <peewee.CharField object>

Deserialize PeerAddress object from given _dict

param _dict:dict containing model attributes and data
type _dict:dict
param **kwargs:extra parameters
returns:deserialized object
rtype:PeerAddress
id = <peewee.AutoField object>
last_seen_at = <peewee.IntegerField object>
location

Location string which is used by clients.

Returns:network location string
Return type:(str)

Parse beiran address

param address:beiran address
type address:str
returns:address details
rtype:(tuple)

Serialize PeerAddress object

param **kwargs:extra parameters
returns:serialized object
rtype:dict
transport = <peewee.CharField object>
uuid = <peewee.UUIDField object>

Validate address

param address:peer address
type address:str
returns:True if address matches pattern, else False
rtype:(bool)

Beiran Libs

Various reusable classes and methods.

Beiran Library

beiran.lib.async_req(url: str, return_json: bool = True, timeout: int = 3, retry: int = 1, retry_interval: int = 2, method: str = 'GET', **kwargs) → Tuple[aiohttp.client_reqrep.ClientResponse, dict][source]

Async http get with aiohttp :param url: get url :type url: str :param return_json: is response json string or not? :type return_json: bool :param timeout: timeout :type timeout: int :param method: HTTP method :type method: str

Returns:response instance, response json
Return type:(ClientResponse, dict)
beiran.lib.async_write_file_stream(url: str, save_path: str, queue=None, timeout: int = 3, retry: int = 1, retry_interval: int = 2, method: str = 'GET', **kwargs) → aiohttp.client_reqrep.ClientResponse[source]

Async write a stream to a file :param url: get url :type url: str :param save_path: path for saving file :type save_path: str :param mode: file mode :type mode: str :param timeout: timeout :type timeout: int :param method: HTTP method :type method: str

Returns:request response
Return type:aiohttp.client_reqrep.ClientResponse

Utilities for beiran project

class beiran.util.Unbuffered(stream: _io.TextIOWrapper)[source]

Bases: object

Unbuffered stream write class

Initialization unbuffered with stream. :param stream: any stream to write unbuffered

Write data to stream and flush :param data: data to write

Write as lines

param lines:array of data
type lines:Array
beiran.util.clean_keys(dict_: dict, keys: list) → None[source]

Remove keys from the dictionary

beiran.util.create_tar_archive(dir_path: str, output_file_path: str)[source]

create a tar archive from given path

Parameters:
  • output_file_path – directory path to be saved!
  • dir_path (string) – directory path to be tarred!

Returns:

beiran.util.eprint(*args, **kwargs)[source]

Printing errors :param *args: :param **kwargs:

beiran.util.exit_print(exit_code: int, *args, **kwargs)[source]

Printing exit code :param exit_code: :param *args: :param **kwargs:

beiran.util.gunzip(input_path: str, output_path: str) → None[source]

Decompress .gz file

beiran.util.input_reader(stream, **kwargs)[source]

input_reder

beiran.util.json_streamer(stream, subpath='$')[source]

Parse a stream of JSON chunks

beiran.util.parse_subpath(subpath)[source]
parse subpath:
This function returns the parsed subpath object. subpath object is just a list.
the regular expression of subpath:
/$(.([w]+|*)|[d*:d*:d*])*/
the grammer of subpath:

subpath ::= “$” selecter* selecter ::= key_selecter | range_selecter key_selecter ::= “.” key_body key_body ::= word* | “*” range_selecter ::= “[” range_body “]” range_body ::= int? “:” int? “:” int?

word ::= “a” | … | “z” | “A” | … | “Z” | digit | “_” digit ::= “0” | … | “9”

ref: http://goessner.net/articles/JsonPath

beiran.util.run_in_loop(coroutine, loop=None, sync=False)[source]

Runs given coroutine in the given or default asyncio loop. Returns Task object is sync if False. If sync is True, blocks the thread and returns the task’s result.

beiran.util.sizeof_fmt(num, suffix='B')[source]

Human readable format for sizes source: https://stackoverflow.com/a/1094933

beiran.util.until_event(emitter: pyee.EventEmitter, name: str, loop=<_UnixSelectorEventLoop running=False closed=False debug=False>)[source]

Wait task until triggered the event

beiran.util.wait_event(emitter, event_name, timeout=None)[source]

Wait until emitter is emitted

beiran.util.wait_task_result(task)[source]

Blocks thread until given task has finished, and returns the result value of the task

Current beiran version constant plus version pretty-print method.

beiran.version.get_version(form: str = 'short', component: str = 'daemon') → Union[dict, str][source]

Return a version string for this package, based on version.

Takes a single argument, form, which should be one of the following strings:

  • branch: just the major + minor, e.g. “0.9”, “1.0”.
  • short (default): compact, e.g. “0.9rc1”, “0.9.0”. For package filenames or SCM tag identifiers.
  • normal: human readable, e.g. “0.9”, “0.9.1”, “0.9 beta 1”. For e.g. documentation site headers.
  • verbose: like normal but fully explicit, e.g. “0.9 final”. For tag commit messages, or anywhere that it’s important to remove ambiguity between a branch and the first final release within that branch.
  • all: Returns all of the above, as a dict.
beiran.version.git_sha()[source]

Git current head sha id

Returns:git sha number
Return type:str

Logging package for daemon

beiran.log.build_logger(filename: str = '/var/log/beirand.log', log_level: str = 10) → logging.Logger[source]

Build logger class for module

Plugin Base

Plugin abstract class for different discovery service implementations.

class beiran.plugin.AbstractBasePlugin[source]

Bases: object

Metaclass for BeiranPlugin modules

Init plugin

Can plugin be utilized by other components at the moment of call

Start plugin

Stop plugin

Sync with another peer node object is accessible at peer.node

class beiran.plugin.BaseDiscoveryPlugin(config: dict)[source]

Bases: beiran.plugin.BasePlugin

Discovery Plugin Base

class DiscoveredNode(hostname: str = None, ip_address: str = None, port: int = None)[source]

Bases: object

Beiran node information class

Initialize self. See help(type(self)) for accurate signature.
address

Gets listen address for daemon

hostname

Gets hostname for discovery

network_interface

Gets listen interface for daemon

port

class beiran.plugin.BaseInterfacePlugin(config: dict)[source]

Bases: beiran.plugin.BasePlugin

Base class for interface plugins

class beiran.plugin.BasePackagePlugin(config: dict)[source]

Bases: beiran.plugin.BasePlugin

Base class for package plugins

class beiran.plugin.BasePlugin(config: dict)[source]

Bases: beiran.plugin.AbstractBasePlugin, pyee.EventEmitter

BeiranPlugin with EventEmmiter

DEFAULTS = {}

Initialization of plugin class with async loop

Emit event, passing *args and **kwargs to each attached function. Returns True if any functions are attached to event; otherwise returns False.

Example:

ee.emit('data', '00101001')

Assuming data is an attached function, this will call data('00101001')'.

For coroutine event handlers, calling emit is non-blocking. In other words, you do not have to await any results from emit, and the coroutine is scheduled in a fire-and-forget fashion.

Init plugin

Initialize plugin configuration. Values ​​not in config are set with default values

Can plugin be utilized by other components at the moment of call

Set dynamic configuration values like using run_dir

Setting log level for plugin classes :param level: logging.level

Start plugin

status

Return plugin status

Stop plugin

Sync with another peer node object is accessible at peer.node

class beiran.plugin.History[source]

Bases: pyee.EventEmitter

Class for keeping update/sync history (of anything)

Initialize self. See help(type(self)) for accurate signature.

Delete updates before time

latest

Latest update or None

Append update to history and increment the version

Return updates since time

beiran.plugin.get_installed_plugins() → List[str][source]

Iterates installed packages and modules to match beiran modules.

Returns:list of package name of installed beiran plugins.
Return type:list

CLI and Clients

CLI

command line client for managing beiran daemon

class beiran.cli.BeiranCLI(name=None, invoke_without_command=False, no_args_is_help=None, subcommand_metavar=None, chain=False, result_callback=None, **attrs)[source]

Bases: click.core.MultiCommand

BeiranCLI loads dynamically commands from installed plugins and appends commands of beiran itself as node group

Load command object

param ctx:context object
type ctx:BeiranContext
param cmd_name:subcommand name
type cmd_name:str
returns:subcommand group object
rtype:click.group

Lists of subcommand names

param ctx:context object
type ctx:BeiranContext
returns:list of subcommand names
rtype:list
class beiran.cli.BeiranContext[source]

Bases: object

Context object for Beiran Commands which keeps clients and other common objects

Initialize self. See help(type(self)) for accurate signature.

Sync Client

Common client for beiran project

class beiran.sync_client.Client(peer_address: beiran.models.node.PeerAddress)[source]

Bases: object

Beiran Client class

Initialization method for client

param peer_address:
 beirand address
type peer_address:
 PeerAddress, str

Get Image list from beiran API :returns: list of images :rtype: list

Get Layer list from beiran API :returns: list of layers :rtype: list

Retrieve information about node :returns: info of node :rtype: object

Daemon get nodes :returns: list of nodes :rtype: list

Gets root path from daemon for server information :returns: parsed from JSON :rtype: object

Daemon version retrieve :returns: semantic version :rtype: str

Retrieve status information about node or one of it’s plugins :returns: status of node or plugin :rtype: object

Connect to a new node :returns: info of node if successful :rtype: object

Pull image accross cluster with spesific node support :returns: Pulling process result :rtype: result

Request call to daemon :param path: http path to request from daemon :param parse_json: if return value is JSON from daemon, :param it returns parsed JSON:

Returns: Response from daemon

class beiran.sync_client.UnixResolver[source]

Bases: tornado.netutil.Resolver

Resolver for unix socket implementation

Closing resolver

Class initialization method :param socket_path: Path for unix socket

Unix Socket resolve :param host: host ip :param port: host port :param family: socket family default socket.AF_UNSPEC :param callback: function to call after resolve

Async Client

Common client for beiran project

class beiran.client.Client(peer_address: beiran.models.node.PeerAddress = None, node: beiran.models.node.Node = None, version: str = None)[source]

Bases: object

Beiran Client class

exception Error(message: str)[source]

Bases: Exception

Base Exception class for Beiran Client operations

Initialize self. See help(type(self)) for accurate signature.
exception HTTPError(status: int, message: str, **kwargs)[source]

Bases: beiran.client.Error

Initialize self. See help(type(self)) for accurate signature.
exception TimeoutError(message: str)[source]

Bases: beiran.client.Error

Initialization method for client :param peer_address: beirand url :type peer_address: PeerAddress :param node: Node (optional) :type node: Node :param version: string (optional) :type version: str

Closes aiohttp client session

Create aiohttp client session

Get Image list from beiran API :returns: list of images :rtype: list

Get Layer list from beiran API :returns: list of layers :rtype: list

Retrieve information about node :returns: info of node :rtype: object

Daemon get nodes :returns: list of nodes :rtype: list

Gets root path from daemon for server information :returns: parsed from JSON :rtype: object

Daemon version retrieve :returns: semantic version :rtype: str

Retrieve status information about node or one of it’s plugins :returns: status of node or plugin :rtype: object

Pings the node

Connect to a new node :returns: info of node if successful :rtype: object

Pull image accross cluster with spesific node support :returns: Pulling process result :rtype: result

Request call to daemon, return successful repsonses or raises http errors.

param path:http path to request from daemon
param data:request payload
type data:dict
param timeout:timeout
type timeout:int
param raise_error:
 raising error
type raise_error:
 bool
param method:request method
type method:str
returns:Response from daemon, dict or json string
rtype:ClientResponse or dict or str
raises convenient http exception, timeout, bad request, etc.:
 

Return json reponse as dict

Return content of response as string

Stream image from this node

Usage:

image_response = await client.stream_image(image_identifier)
async for data in image_response.content.iter_chunked(64*1024):
    do something with data chunk