Validator Devops contest for Rust-Node

Hi folks!
After contest discussion on the DevOps subgov weekly call (join as if you still not in the group - > https://t.me/freetondevops) we have two suggestions:

  1. Extend contest rewards pool but add vesting in equal parts over a 12-month period with monthly payouts. To receive all 12 distributions, the solution should be supported for 1 year
    The scripts should be updated within a reasonable timeframe - 1 month maximum, and according to rust node changes
    Major GitHub Issues should be resolved within a reasonable timeframe - 1 month maximum.
    Winners receive 30% of their rewards when contest results are evaluated. I propose this reward distribution table:
    1st prize — 40,000 Tons (12 000 + 28 000 by vesting)
    2nd prize — 30,000 Tons (9 000 + 21 000 by vesting)
    3rd prize — 20,000 Tons (6 000 + 14 000 by vesting)
    4th prize — 5,000 Tons
    5th prize — 3,000 Tons
    Next 10 runners up — 1,000 Tons each
  2. Add minimum of completed tasks (task 1, and task 4 or 5) (without them submission can be rejected) to simplify evaluation creteria and voting. Otherwise juries will potentially compare different types of works (some will be focus on monitoring/alerting, some on installation/validation scripts) which will hard and incorrect to compare (because of different purpose)

Let’s discuss. Please agree/disagree.

5 Likes

I want to clarify, imho
for task 1
for the minimum of the task completion will enough for one OS

1 Like

Good. Will amend. Should change the dates as well.

Just double-checked, according to the DevOps SG proposal (voted and passed), the reward for this contest is supposed to be 200’000, not 103’000. Compared to the first C++ DevOps node contest, minimum number of tasks in this one is mandated (i. e. more work required). For the C++ node DevOps node contest, the rough equivalent of task 3 has been done by TonLabs (scripts - validator_msig.sh, validator_depool.sh), and task 4 hasn’t been there at all (would’ve been something like automating staking through depool using liteclient). Other than that, requiring MVP (minimum number of tasks completed) and vesting for the first 3 places is a good idea.

Short description

Monitoring, Alerting, Scripts and Tooling for RUST implementation of Free TON Validator

Type

Contest

Dates

Submissions - Starts: 18.01.2021 at 00:00 UTC, Ends: 31.01.2021 at 23:59 UTC

Voting - Starts: 01.02.2021 at 00:00 UTC, Ends: 14.02.2021 at 23:59 UTC

Motivation

Validators need tools and automation to insure efficient scalable and secure operations to support Free TON blockchain

General requirements:

  • Documentation should be provided for each tool
  • Provided tools should operate exactly according to its documentation
  • Motivation and goal for each tool should be described
  • All software must be released with licences supported by Free Software Foundation

DevOps Skills required:

  • Linux
  • Unix Shell scripting, javascript
  • Infrastructure as a code
  • docker/docker-compose/kubernetes
  • Desirable: Ansible, logstash, Prometheus/Grafana

Tasks:

  1. Enhanced node deployment infrastructure
    • Need to develop enhanced deployment infrastructure for Rust validator nodes based on modern tools and techniques (docker/docker-compose, Ansible, Terraform, etc.).
    • Based on infrastructure as a code. Take into account deployment from scratch, upgrade, cleanup scenarios on various set of Linux distros as well as other OS (e.g. MacOS).
  2. Intellectual log dashboard with analytics:
    • Need to collect log files into centralised database (e.g. logstash).
    • Make base analytics with some predefined patterns which can simplify failure analysis and monitoring.
  3. Extended metrics for node monitoring
    • Analyse available node/validator/elector metrics and define a subset useful for further analytics and monitoring tasks.
    • Develop visualisation concept to simplify the analysis.
    • Develop alert rules for daily monitoring
  4. Automate staking through depool (and/or direct staking to the elector) based on tonos-cli or SDK (use TON-SDK core library or any existing binding)
    • Write validator script for elections using tonos-cli/SDK
    • Use tonos-cli/SDK to run elector get-methods.
    • Use tonos-cli/SDK to query global configuration parameters.
    • Use tonos-cli/SDK to prepare, sign and send validator query to depool/elector smart contract.
    • Automate confirmation of multisig transactions.
    • Improve custodian notification mechanism for multisig transactions: notify about submitted transactions via user-friendly mechanism (e-mail, sms, etc.).
    • Implement tools for staking monitoring and alerting
  5. Automate staking through depool (and/or direct staking to the elector) using R-Node Console
    • Write a script to manage overall staking procedure (prepare elector request, sign, send, manage bids, etc.)
    • Automate confirmation of multisig transactions
    • Implement tools for staking monitoring and alerting

Evaluation criteria and Winning conditions:

  • Number of completed tasks. Tasks 1 (at least for one OS) and optionally 4 or 5 are necessary. Without them submission will be rejected
  • Correctness of operations of the provided tool
  • Quality of code and description
  • Work as advertised, perform its stated functions

The Jury

The Jury will be selected by a sub-governance which will support this contest.

Jury rewards:

An amount equal to 5% of the sum total of all total tokens actually awarded to winners of this contest will be divided equally between all jurors who vote and provide feedback. Both voting and feedback are mandatory in order to collect this reward.

Winners:

The winners of all 5 tasks will recieve the following:

1st prize — 40,000 Tons (16 000 + 24 000 by vesting)
2nd prize — 30,000 Tons (12 000+ 18 000 by vesting)
3rd prize — 20,000 Tons (8 000 + 12 000 by vesting)
4th prize — 5,000 Tons
5th prize — 3,000 Tons
6-10th price — 1,000 Tons each

Winners receive 40% of their rewards when contest results are evaluated.
Another 60% will be vested over a 12-month period with monthly payouts.

Vesting details

Vesting is devided into 12 equal parts. The first vesting payout is paid one month after the end of the contest. To receive all 12 distributions, the solution should be supported at least for 1 year. Scripts should be updated within a reasonable timeframe - 1 month maximum, and according to rust node changes. Major GitHub Issues should be resolved within a reasonable timeframe - 1 month maximum.

2 Likes

SG members decided to move remaining costs to future contests.

Let me introduce build&run tools for the rust node.

Currently they allows to build and run the rust node on the rustnet.

Tested on Ubuntu 20.04

Please leave your feedback at tg: @asddsa1137

UPD

  1. Fixed many issues.
  2. Added sync status script.
  3. Added validator scripts both for single and depool staking.

Please note that currently rust node from master branch, as well as rustnet, are kind of broken.

4 Likes

Cool! Thank you! Anyway devops sg plan to prolong this contest.

UPD

  1. Fixed many issues.
  2. Added sync status script.
  3. Added validator scripts both for single and depool staking.

Please note that currently rust node from master branch, as well as rustnet, are kind of broken.

5 Likes

Solution in GitHub - itgoldio/freeton-rustnode-ansible: Ansible playbooks and scripts for install and manage freeton rust node

Include:

  1. ansible playbook for build and install Rust-Node, totonos-cli, ton-labs-node-tools, TVM-linker.
  2. monitoring (optional for installation)
    2.1 telegraf for collect metrics
    2.2 kibana for visualisate
    2.3 influxdb for store
  3. scripts
    3.1 for staking throught depool
    3.2 for ticktok depool
    3.3 for confirmation transaction
    and more.

Support:

  1. github issue
  2. telegram chat en Telegram: Contact @tgoldio_support_en
  3. telegram chat ru Telegram: Contact @itgoldio_support_ru

Plans

  1. add more scripts
  2. make dashboards better
  3. add scripts for validation without depool
  4. improve readme
3 Likes

Hello we are happy to introduce our solution to deploy and monitoring rust node
https://github.com/stack2me/ansible-freeton-rust

Functionality

  • Install rust node
  • Creating user and group
  • Cronjob for validator script
  • All logs in one folder /var/ton/log/
  • All node configs, keys, contracts in /etc/ton
  • Systemd for control status of node and restart in fail case
  • Logrotate and archive logs
  • Node Monitoring
    • Notification to telegram chat if diff < -50 and if balance < 10001
    • Install netdata for realtime status /netdata
    • Install prometheus-node-exporter for collect metrics
      • collecting data about node status:
        • node diff
        • wallet balance
        • total validators
        • node validation status
        • elections status
      • Install nginx for set basic auth for entry points of monitoring systems
  • Notifications about node events like an open election, approve/reject transactions etc…
  • Script control transactions (approve/reject) with notifications to telegram
  • Install and sync ntp server for avoid time shift
  • Install system upgrade

our contact:
site: https://stack2.me
email: stack2me@protonmail.com
telegram: @beardr3d

2 Likes

Framework for managing and controlling validator with infrastructure-as-a-code approach.

(note: in submission made typo in the telegram name, correct is @jar1g).


Key Points:

  • Validator node doesn’t have any extra ports exposed
  • Every deployment can be scaled independently and whenever is required
  • Very flexible in controlling costs - Validator, Controller and Logstash are deployed via Docker (backed-up with docker-compose) either to bare-metal machine or VM. At the same time monitoring can be either custom solution or one of SaaS solutions with pay-as-you-go subscriptions. The same applies for message-queue (either custom deployment or SaaS).
  • Pub/Sub layer provides good abstraction and allows to inject many type of notifications and ways to control validator(s), including safe for the validator web interfaces.
  • It is easy to integrate any kind of alerting and automatic response to those alerts.

Has following features:

  • Deploy, execute and monitor validator node(s) using infrastructure-as-a-code approach. So, sharing setup is seamless and easy across team members.
  • Fully automatic participation in elections (direct or depool), including automatic confirmations for custodians.
  • Can participate in elections with check of potential win in the elections. So, for example, if a node has small chance of being elected (stake that can be made is too small to be among participating nodes), then election can be skipped. All thresholds are configurable.
  • Reporting telemetry via logstash to any configured ELK stack service. Can be use with such SaaS as Bonsai.io, no need to deploy own solution (but for sure possible).
2 Likes

All scripts are in the active development stage and will be updated regularly. Feel free to contact as on github thought issues or directly on Telegram. Scripts done by validators for validators.

https://github.com/INTONNATION/FreeTON-Rust-Node

@Gofman @renatSK
@sostrovskyi @sostrovskyi

1 Like

Here’s my submission, too.

I’ll galdly answer any related questions, if you ping me via Telegram (@sergemedvedev).

2 Likes

Please show screenshots of Grafana and Chronograf

Hello, i’m from phone today, I can insert screenshots to this post tomorrow and i will add it to readme tomorrow. Can you wait it, or it important?

Hello, i added screenshots to github repo


Grafana dashboard:

  • show current timediff and timediff graph
  • Current Election state (Started/Stopped)
  • Current Election date start (date or undefine if elector don’t started)
  • Current Election date stop (date or undefine if elector don’t started)
  • Info, that node already in participant list
  • info, that node already win election and wait new validation cycle
  • count of unsign transaction between wallet to depool
  • Wallet balance
  • depool balance
  • proxy-1 balance
  • proxy-2 balance
  • many system graphs, we change it after testing node in rustcup
3 Likes

What happened to the contest winner’s results? Are they added in some official release? It looks as they won, but, their work hasn’t been implemented anywhere.

Where they should be implemented?

There is nothing official in Free TON

1 Like