# Open Platform Introduction

KingData Open Platform is a user-defined indicators layer, built by the community.

KingData is a platform focusing on data monitoring and broadcasting in the Web3. Currently, the platform has opened 500+ indicators for users to choose, covering various subdivisions, and is determined to become a necessary data monitoring platform for Web3 users. Let industry users not miss any data fluctuations that they care about.

In order to provide more valuable data change services to industry users, KingData has opened an open platform for indicator development. Any developer can easily write their favorite data change monitoring indicators according to the specifications of this open platform. 

KingData Open Platform are recipes to build high level indicators that users care about most, like large NFT Trade, large transfers of Token etc. Indicator write in Python.

All indicator requirements are managed through [https://dework.xyz/](https://app.dework.xyz/kingdata-17178). Here we can find all indicators currently in development.

How to get started building on KingData and create data monitoring that you care about. Next, we demonstrate one by one.

Developers can set their own indicators to be VIP or Free.

#### Free

    Everyone can follow and use it for free. After following, they will automatically receive the data broadcast.

#### VIP

1. Users need to pay to become a platform VIP before they can use follow.
2. Developers of VIP indicators can get 50% revenue sharing from VIP indicators every month.

Most Indicators on KingData are contributed and maintained by communities, and all changes are coordinated through the <https://github.com/js-kingdata/indicators_factory> GitHub repo.

## Next Steps


# How to develop an indicator

Getting Started Instructions

{% hint style="info" %}

<pre data-overflow="wrap"><code>It is recommended to read our <a data-footnote-ref href="#user-content-fn-1">FAQ</a> and Data Storage Solutions first. so as to be familiar with various problems that need to be dealt with during the development process.
</code></pre>

{% endhint %}

Steps to develop indicators on KingData:

1. Fork the [indicators repository](https://github.com/js-kingdata/indicators_factory) (button towards the top right of the repo page).&#x20;
2. Add a new folder in `/crawlers/indicators/spiders/` with the new indicator name.&#x20;
3. Write the indicator's description file in the new folder: `definition.md`.&#x20;
4. Create a new crawler file in this directory to realize the indicator requirement logic of `definition.md`.&#x20;
5. Write broadcast template to render the params that logic code return.
6. Test and submit the changes on your fork to the [indicators repository](https://github.com/js-kingdata/indicators_factory) as a Pull Request with a brief description of the changes you made.&#x20;
7. Waiting for someone to comment or merge your merge request. There is no need to have someone check your PR as it is regularly monitored.&#x20;
8. Once your PR is merged, please give the front-end team 24 hours to load your listing on the UI.

## Indicators development environments

1. Development Language：Python 3.10.4, Our recommendation is to follow the [Hitchhiker's Guide to Python](https://docs.python-guide.org/starting/installation/).
2. Crawler framework：Scrapy 2.7.1  Our recommendation is to follow the  [Guide to Scrapy](https://docs.scrapy.org/en/latest/intro/tutorial.html).&#x20;

[^1]:


# Step1: Clone Repo

Make a [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of the [Repo](https://github.com/js-kingdata/indicators_factory). Including cloning locally and adding an upstream.

Review Github’s [instructions](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) on how to make a pull request from a fork.

After that, you will get the following directory structure：

![](https://1423041546-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhRdsuMamDl5gKestkuSr%2Fuploads%2F6iMrqRXxq2BJjLyY0AVc%2Fimage.png?alt=media\&token=5df53018-f6c6-4bc8-b2cd-51f3e5717421)    &#x20;

### crawlers/indicators/spiders

One indicator corresponds to one file, and the original files of all indicators are placed in it.

### crawlers/utils

Many public methods that crawlers need to use are placed here. Such as data formatting, Redis tools, etc.

## Create new indicator

We need create a new folder in `crawlers/indicators/spiders`

For Example: `mainstream_coin_long_short_ratio.`

This folder needs to contain thress basic files:

* Definition.md - This file contains background information about the indicator.
* indicator\_name.py - This file contains indicator logic code.
* \_\_init\_\_.py - This file is to config your inidicator spider.


# Step2: Write Definition.md

Definition.md describes the full requirements background of the current indicator

This document describes in detail the requirement background of the indicators to be created, as well as the relevant statistical broadcast rules. It is mainly described from the following aspects.

Create a new `Definition.md` file in the directory `crawlers/indicators/spiders/mainstream_coin_long_short_ratio`. The structure and content are as follows, and fill in the content according to the requirements

## Definition.md structure

```
## Support Language (Required)


## Indicator Name (Required)


## Why Create This Indicator (Optional)


## Indicator Category (Optional)


## Free or VIP (Required)


## Indicator Chain (Optional)


## Indicator Data Source (Optional)


## Indicator Alert Frequency (Required)


## Indicator Alert logic (Required)


## Indicator Alert Demo (Required)
```

## Support Language

*Options: EN, CN, EN & CN*

For example：`EN & CN`

## Indicator Name

*This name will be displayed on the KingData UI, and users will judge whether they are interested based on this name.*

*If the indicator support two language, you need to provide names in both languages*

For example：`Mainstream Coin Long/Short Ratio`

## Why Create This Indicator

*Make it clear why you want to create the indicator,  who will use it.*

For example：When the market changes, I want to know the distribution of long and short people in the current BTC markets at the earliest. Users who trade in the secondary market, especially those who trade unilaterally, pay close attention to this indicator.

## Indicator Category

*The category this Indicator belongs to.*

For example：`DeFi、CEX、On-chain etc.`

## Free or VIP

*Do you want the indicator to be free for users or for a fee?*

*Free is everyone can subscribe this indicator.*

*VIP require users to pay to subscribe.*

For example：Free`.`

## Indicator Chain

*The indicator belongs to the public chain, if not, fill in: None*

For example：`ETH`

## Indicator Data Source

*Describe the data source for this indicator*

For example：<https://www.coinglass.com/zh>

## Indicator Alert Frequency

Indicates how often this indicator is broadcast, or how often data is checked

For example：`broadcast every 4 hours`

## Indicator Alert Logic

Describe in detail the data broadcast processing logic of this indicator

For example：1H BTC price rises and falls by more than 3% and broadcasts the liquidation data / 24H, BTC rises and falls by more than 7% and broadcasts the liquidation data.

## Indicator Alert Demo

Provide broadcast text in Chinese and English. After the user pays attention to the indicator, if there is a data change in the indicator, the user will receive an indicator broadcast with similar content.

For example：

EN:

```
According to KingData monitoring, In the past 4 hours, BTC Long/Short Ratio across network is 0.67, with 39.99% longs and  60.01% shorts. Shorts outweigh Longs.
Among leading exchanges:
Binance: long 41.21%, short 58.79%
OKX: long 38.17%, short 61.83%
Bitget: long 45.42%, short 54.58%

```

CN:

```
据 KingData 数据监控，最近4小时，BTC全网合约多空比为 0.67，多单占比 39.99%，空单占比60.01%，看空人数大于看多人数。
其中主流交易所：
Binance：做多41.21% 做空58.79%
OKX：做多38.17% 做空61.83%
Bitget：做多45.42% 做空54.58%

```

## Other Instructions

Other additional instructions...


# Step3: Write Logic Code

To achieve an indicator, we usually consider the following issues

1. What data source is used&#x20;
2. What is the processing logic and how to set the threshold&#x20;
3. Output parameters&#x20;
4. Render template

Create a new `new_coin_spider.py` file in the directory `crawlers/indicators/spiders/mainstream_coin_long_short_ratio`. The code are as follows

```python
from crawlers.utils import SpiderBase
from jinja2 import Template
from crawlers.utils.group_alarm import catch_except


def build_coin_info(data):
    return {
        'symbol': data['symbol'],
        'long_short_rate': round(data['longRate'] / data['shortRate'], 2),
        'long_rate': data['longRate'],
        'short_rate': data['shortRate'],
        'list': [{
            'exchange_name': exchange_info['exchangeName'],
            'long_rate': exchange_info['longRate'],
            'short_rate': exchange_info['shortRate']
        } for exchange_info in data['list'][:3]]
    }


class ContractPositionRatio(SpiderBase):
    name = 'idx-contract-position-ratio'

    start_urls = [
        'https://fapi.coinglass.com/api/futures/longShortRate?symbol=BTC&timeType=3',
        'https://fapi.coinglass.com/api/futures/longShortRate?symbol=ETH&timeType=3'
    ]

    @catch_except
    def parse(self, response, **kwargs):
        data = response.json()['data'][0]
        params = {
            'info': [build_coin_info(data)]
        }
```

Next, we need to define the template of the broadcast, and the template is used to render the variables in params


# Step4: Write Template

The template can clearly explain the alert data, making it easier for users to understand.

Because the data fields referenced in each indicator are different, there is no reusability. In order to improve the flexibility of the data and make the expression of the alert more understandable, we have separated the processing of the indicator data from the broadcast text, and introduced the template engine. We create an alert template corresponding to the language for each indicator, and the data that needs to be used in the template is replaced by variables. After the logic of the indicators is processed, the corresponding data variables are passed to the template, which can be easily compiled into broadcast text in different languages.

We are using [`Jinjia2`](https://jinja.palletsprojects.com/en/3.0.x/)   templating engine

Jinja is a fast, expressive, extensible templating engine. Special placeholders in the template allow writing code similar to Python syntax. Then the template is passed data to render the final document.   More info:  <https://github.com/pallets/jinja>

## For Example

The corresponding EN template is:

{% code overflow="wrap" %}

```django
According to KingData monitoring, {% for coin in info%}In the past 4 hours, {{coin.symbol}} Long/Short Ratio across network is {{coin.long_short_rate}}, with {{coin.long_rate}}% longs and  {{coin.short_rate}}% shorts. {%if coin.long_short_rate>1%}Longs outweigh shorts{% else %}Shorts outweigh Longs{% endif %}.
Among leading exchanges:{% for exchange in coin.list %}
{{exchange.exchange_name}}: long {{exchange.long_rate}}%, short {{exchange.short_rate}}%{% endfor %}

{% endfor %}
```

{% endcode %}

The rendered EN text is：

{% code overflow="wrap" %}

```
According to KingData monitoring, In the past 4 hours, BTC Long/Short Ratio across network is 1.08, with 51.86% longs and  48.14% shorts. Longs outweigh shorts.
Among leading exchanges:
Binance: long 46.4%, short 53.6%
OKX: long 57.96%, short 42.04%
Bitget: long 50.53%, short 49.47%

```

{% endcode %}

The corresponding CN template is:

{% code overflow="wrap" %}

```django
据 KingData 数据监控，{% for coin in info%}最近4小时，{{coin.symbol}}全网合约多空比为 {{coin.long_short_rate}}，多单占比 {{coin.long_rate}}%，空单占比{{coin.short_rate}}%，{%if coin.long_short_rate>1%}看多人数大于看空人数{% else %}看空人数大于看多人数{% endif %}。
其中主流交易所：{% for exchange in coin.list %}
{{exchange.exchange_name}}：做多{{exchange.long_rate}}% 做空{{exchange.short_rate}}%{% endfor %}

{% endfor %}
```

{% endcode %}

The rendered CN text is：

{% code overflow="wrap" %}

```
据 KingData 数据监控，最近4小时，BTC全网合约多空比为 1.08，多单占比 51.86%，空单占比48.14%，看多人数大于看空人数。
其中主流交易所：
Binance：做多46.4% 做空53.6%
OKX：做多57.96% 做空42.04%
Bitget：做多50.53% 做空49.47%

```

{% endcode %}

## So the complete code is

```python
from crawlers.utils import SpiderBase
from jinja2 import Template
from crawlers.utils.group_alarm import catch_except


def build_coin_info(data):
    return {
        'symbol': data['symbol'],
        'long_short_rate': round(data['longRate'] / data['shortRate'], 2),
        'long_rate': data['longRate'],
        'short_rate': data['shortRate'],
        'list': [{
            'exchange_name': exchange_info['exchangeName'],
            'long_rate': exchange_info['longRate'],
            'short_rate': exchange_info['shortRate']
        } for exchange_info in data['list'][:3]]
    }


class ContractPositionRatio(SpiderBase):
    name = 'idx-contract-position-ratio'

    start_urls = [
        'https://fapi.coinglass.com/api/futures/longShortRate?symbol=BTC&timeType=3',
        'https://fapi.coinglass.com/api/futures/longShortRate?symbol=ETH&timeType=3'
    ]

    @catch_except
    def parse(self, response, **kwargs):
        data = response.json()['data'][0]
        params = {
            'info': [build_coin_info(data)]
        }
        print(Template(self.alert_en_template()).render(params))
        print(Template(self.alert_cn_template()).render(params))

    # must be declare
    def alert_en_template(self):
        return """
According to KingData monitoring, {% for coin in info%}In the past 4 hours, {{coin.symbol}} Long/Short Ratio across network is {{coin.long_short_rate}}, with {{coin.long_rate}}% longs and  {{coin.short_rate}}% shorts. {%if coin.long_short_rate>1%}Longs outweigh shorts{% else %}Shorts outweigh Longs{% endif %}.
Among leading exchanges:{% for exchange in coin.list %}
{{exchange.exchange_name}}: long {{exchange.long_rate}}%, short {{exchange.short_rate}}%{% endfor %}
{% endfor %}
"""

    # must be declare
    def alert_cn_template(self):
        return """
据 KingData 数据监控，{% for coin in info%}最近4小时，{{coin.symbol}}全网合约多空比为 {{coin.long_short_rate}}，多单占比 {{coin.long_rate}}%，空单占比{{coin.short_rate}}%，{%if coin.long_short_rate>1%}看多人数大于看空人数{% else %}看空人数大于看多人数{% endif %}。
其中主流交易所：{% for exchange in coin.list %}
{{exchange.exchange_name}}：做多{{exchange.long_rate}}% 做空{{exchange.short_rate}}%{% endfor %}
{% endfor %}
"""

```


# Step5: Test and Pull Request

So far we have completed all the work that needs to be opened for an indicator. Next, we test our crawler and submit a pull-request.

## Test Our Crawler

Change to the root directory of the project

```shell
$ scrapy crawl idx-contract-position-ratio
```

`idx-contract-position-ratio` is the name of the crawler, which we defined in the code.

{% hint style="success" %}
If the corresponding template  content can be printed normally according to our expectations, the test is passed
{% endhint %}

## Pull Request

The only thing to do next is to submit a pull-request on Github, and then wait for the review of the staff.


# Examples

In order to better write the crawler code, 3 additional Demos are provided. You can learn the processing logic from the corresponding code.

1. Crawler Demo with JSON Data
2. Crawler Demo with HTML Data
3. Crawler Demo with Socket Data


# Demo with JSON data

Demonstrate what to do if the data source we crawl is JSON

### Indicator :  BTC-ARH999

```python
import scrapy

from crawlers.utils import SpiderBase, Tools
from crawlers.utils.group_alarm import catch_except
from jinja2 import Template


class BtcArh999Spider(SpiderBase):
    name = 'idx-btc-arh999'
    url = 'https://fapi.coinglass.com/api/index/ahr999'

    def start_requests(self):
        yield scrapy.Request(url=self.url)

    @catch_except
    def parse(self, response, **kwargs):
        data = response.json()['data']
        params = {
            'arh_999': round(data[-1]['ahr999'], 2),
            'btc_price': data[-1]['value'],
            'change': round(((float(data[-1]['value']) - float(data[-2]['value'])) / float(data[-2]['value'])) * 100, 2)
        }
        print(Template(self.alert_en_template()).render(params))
        print(Template(self.alert_cn_template()).render(params))

    # must be declare
    def alert_en_template(self):
        return """The current BTC ahr999 (AHR Index) is {{arh_999}}. This spot is theoretically unsuitable for bottom fishing or long-term fixed investment. The current price of BTC is {{btc_price}}, and 24H  change is {{change}}. (The above content does not constitute investment advice and is for your reference only. Invest at your own risk.)
        """

    # must be declare
    def alert_cn_template(self):
        return """当前 BTC ahr999(九神指数)为 {{arh_999}}，理论上不宜买入抄底或定投 BTC。当前 BTC 现价 {{btc_price}}，24小时涨跌幅为 {{change}}。（以上内容仅供参考，非投资建议，风险自担。）
        """

```


# Demo with HTML data

Demonstrate what to do if the data source we crawl is HTML

### Indicator:  BTC MVRV Alert

```python
import scrapy
import re

from crawlers.utils import SpiderBase, Tools
from crawlers.utils.group_alarm import catch_except
from jinja2 import Template


def minimal_regularity(pattern, string):
    comp = re.compile(pattern, re.S)
    return re.findall(comp, string)


class BtcMVRVSpider(SpiderBase):
    name = 'idx-btc-mvrv'
    url = 'https://charts.woobull.com/bitcoin-mvrv-ratio/'

    def start_requests(self):
        # error_back method is defined in SpiderBase
        yield scrapy.Request(url=self.url, errback=self.error_back)

    # Exceptions must be handled, only need to be declared, and the processing logic system has been processed uniformly
    @catch_except
    def parse(self, response, **kwargs):
        string = response.text
        for pattern in ['mvrv = .*?}', 'y:.*?]', r'\[.*\]']:
            # Get the desired data from html
            string = minimal_regularity(pattern, string)[0]
        btc_mv_rv = eval(string)[-1]
        params = {
            'mv_rv': round(btc_mv_rv, 2)
        }
        print(Template(self.alert_en_template()).render(params))
        print(Template(self.alert_cn_template()).render(params))

    # must be declare
    def alert_en_template(self):
        return """According to KingData monitoring, BTC current MVRV ratio is {{mv_rv}}，{% if mv_rv < 1 %}theoretically the price is at bottom, marking late stage bear market accumulations.{% endif %}{% if mv_rv > 3.7 %} theoretically the price is at top, signaling late stage bull cycles.{% endif %}{% if 1<= mv_rv <= 3.7 %}Theoretically, MVRV values over '3.7' indicated price top and values below '1' indicated price bottom.{% endif %}
 (The above content is for your reference only and does not constitute investment advice. Invest at your own risk.)  """

    # must be declare
    def alert_cn_template(self):
        return """据 KingData 数据监控，BTC 当前 MVRV 比率为 {{mv_rv}}，{% if mv_rv < 1 %}理论上价格见底，市场处在熊市晚期积累阶段。{% endif %}{% if mv_rv > 3.7 %}理论上价格见顶，市场处在牛市后期。{% endif %}{% if 1<= mv_rv <= 3.7 %}理论上，MVRV > 3.7 是价格见顶信号，MVRV < 1 是价格见底信号。{% endif %}
（以上内容仅作参考，不构成投资建议，风险自担。）"""
```

{% code overflow="wrap" %}

```python
```

{% endcode %}


# Demo with Socket Data

Sometimes, we have high requirements for the timeliness of the indicators. At this time, we often use  socket connections with the data source, such as market price alert. In KingData, the startup of the socket is not deployed through Scrapyd, so it is slightly different, but Users only need to expose the startup method, and KingData will automatically inherit other development work into existing services. It should be noted that we need to handle various abnormal operations such as connection timeout, heartbeat, and reconnection in the program.

### Indicator :  BTC Price Alert

```python
import typer
import asyncio
import aiohttp
import ujson

async def binance_conn():
    websocket = await _conn('wss://stream.binance.com:443/ws/!miniTicker@arr')
    return websocket
    
async def _conn(uri):
    session = aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(100))
    websocket = await session.ws_connect(uri, proxy=PROXY, ssl=False)
    return websocket
    
async def get_k_line(kline_q):
    print('binance web Socket Connecting ...')
    websocket = await ws.binance_conn()
    ts = time.time()
    print('binance web Socket Connect Success!')

    while 1:
        try:
            msg = await websocket.receive(timeout=10)
        except asyncio.exceptions.TimeoutError as e:
            print('TimeoutError: Reconnecting')
            websocket = await ws.binance_conn()
            await asyncio.sleep(1)
        else:
            match msg.type:
                case aiohttp.WSMsgType.TEXT:
                    data = ujson.loads(msg.data)
                    for ticker in data:
                        # TODO Real-time data logic processing
                        
                case aiohttp.WSMsgType.CLOSED:
                    print('Binance web Socket Connect Failed!')
                    websocket = await ws.binance_conn()
                    await asyncio.sleep(1)

===== starting program =====

app = typer.Typer()
async def run():
    kline_q = asyncio.queues.Queue(maxsize=10000)
    await asyncio.gather(
        get_k_line(kline_q)
    )

@app.command()
def start():
    asyncio.run(run())

@app.command()
def goodbye(name: str, formal: bool = False):
    if formal:
        typer.echo(f"Goodbye Ms. {name}. Have a good day.")
    else:
        typer.echo(f"Bye {name}!")

if __name__ == "__main__":
    app()        
```

## Starting Program

```shell
$ python ./btc_alert.py start
```


# ⚠️ Attention && FAQ

## Historical data does not support storage

Some indicators often have a lot of historical trend data, which currently do not support storage. For example: When the USDT/USDT premium is greater than 1%, notify me！ we only need to store and update the latest processed data in Redis, and do not store historical data. So you need to pay attention: when the indicator is used for the first time, there is no historical data in Redis for reference and comparison. In the logic processing, it is necessary to do a good job of data initialization. Avoid program exceptions.

## Data content translation

If the data content needs to be translated in the indicators, for example, if we crawl exchange announcements, some are in English and some are in Korean, then we need to translate the content into Chinese or English, and more other language versions. This is required to use translation services. The <https://deepl.com> service currently used by KingData. Of course, you can also choose other services. It can be implemented according to your indicator logic.

## Public method

Before writing code, you can check all the files under `crawlers/utils`, which abstract a large number of public methods. If you find that there is no public method that meets your needs, just create a new corresponding method or file in the directory.

## Proxy server

If the crawler needs a proxy service, the proxy can be ignored during local development, and KingData has a unified proxy service. You only need to declare in the `Definition.md` file that you need to use the proxy service.

## Exception handling

Each crawler will inevitably have various exceptions in parsing and processing data logic, so `@catch_except` must be declared in the parsing logic method (there is a demonstration in the Demo). After the declaration, when an exception occurs in the program, the system will automatically broadcast the exception to the system's alarm platform.

## Broadcast frequency limit

Even if a processing threshold is set for each indicator, sometimes a large amount of content may be broadcast. If there are a large number of high-frequency pushes, will it cause trouble for users who follow this indicator? Don't worry about this, the KingData backend limits the overall broadcast frequency. Users will not receive a large number of pushes in a short period of time.


# How to store data

Data storage tutorial

Some indicators need to refer to previous data as a reference when processing logic, such as the time of last processing, the result of last data processing, etc. Here we need to store the data in the process or the result data for subsequent use.

Currently, DB-level data storage is not supported, and support for Redis-level data storage is currently provided.

## Set up a Redis connection

When testing and developing locally, set `REDIS_URL = env.str('REDIS_URL', 'redis://127.0.0.1:6379/0')`. in the `crawlers/config.py` file to the local RUL to use.

## Redis usage

Some commonly used Redis operations are defined in the `uitls/redis_conn.py` file of the project library.

At present, only three methods are provided externally, and only three methods are supported, as shown in the following code:

```python
import logging

import redis
import time

from crawlers.config import REDIS_URL
from urllib.parse import urlparse

_REDIS_URL = urlparse(REDIS_URL)

pool = redis.connection.ConnectionPool(
    host=_REDIS_URL.hostname,
    port=_REDIS_URL.port,
    db=_REDIS_URL.path[1:],
)

_redis_client = redis.Redis(connection_pool=pool)


class rds:

    @classmethod
    def getex(cls, prefix, name):
        """
           Return the value at key ``prefix + ':' + name``, or None if the key doesn't exist
           prefix: The prefix parameter indicates the name value of the current crawler
           name: Customize the name value related to the current business
        """
        key = prefix + ':' + name

        if len(key.encode()) > 1024:
            logging.warning('Key is too large')
            return None
        value = _redis_client.get(key)
        if value:
            return str(value, encoding="utf-8")
        return value

    @classmethod
    def setex(cls, prefix, name: str, value: str, ttl):
        """
           Return the value at key ``prefix + ':' + name``, or None if the key doesn't exist
           prefix: The prefix parameter indicates the name value of the current crawler
           name: Customize the name value related to the current business. The key string size cannot exceed 1KB
           value: The stored value cannot exceed 128 KB
           ttl: Expiration time must be set, and value must be taken according to business requirements
        """
        key = prefix + ':' + name
        # Size limit, value cannot exceed 128 KB, key cannot exceed 1 KB
        if len(value.encode()) > 1024 * 128 or len(key.encode()) > 1024:
            logging.warning('Key or Value is too large')
            return False

        if _redis_client.get(key) is not None:
            return False

        if ttl:
            _redis_client.set(key, value, ex=ttl)
        else:
            _redis_client.set(key, value)
        return True

    @classmethod
    def get_and_set_key(cls, prefix, name: str, value: str, ttl: int = None):
        """
            Return the value at key ``prefix + ':' + name``, or True if the key exist
            prefix: The prefix parameter indicates the name value of the current crawler
            name: Customize the name value related to the current business. The key string size cannot exceed 1KB
            value: The stored value cannot exceed 128 KB
            ttl: Expiration time must be set, and value must be taken according to business requirements
         """
        key = prefix + ':' + name
        # Size limit, value cannot exceed 128 KB, key cannot exceed 1 KB
        if len(value.encode()) > 1024 * 128 or len(key.encode()) > 1024:
            logging.warning('Key or Value is too large')
            return False

        if _redis_client.get(name):
            return True
        _redis_client.set(name, value)
        if ttl:
            _redis_client.expire(name, ttl)

    @classmethod
    def thing_lock(cls, name, expiration_time=2, time_out=3):
        """
        code pessimistic locking
        Function: Avoid simultaneous execution of functions, resulting in unpredictable problems
        """
        def outer_func(func):
            def wrapper_func(*args, **kwargs):
                lock_name = f'lock:{name}'
                end_time = time.time() + time_out
                while time.time() < end_time:
                    if _redis_client.setnx(lock_name, expiration_time):
                        _redis_client.expire(lock_name, expiration_time)
                        data = func(*args, **kwargs)
                        _redis_client.delete(lock_name)
                        return data
                    time.sleep(0.001)
                return func(*args, **kwargs)

            return wrapper_func

        return outer_func





```

## Todo:

The prefix field is the prefix of the Redis key and is taken from the name value of the spider.

<figure><img src="https://1423041546-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhRdsuMamDl5gKestkuSr%2Fuploads%2FWxPeIAxTiqyZQb3YCqqh%2Fimage.png?alt=media&amp;token=8593a8a5-a20b-4575-a0d3-68fd088fbada" alt=""><figcaption></figcaption></figure>

## Demo

Set value:

```python
import json

import scrapy
import time
import datetime
from crawlers.utils import SpiderBase, rds
from crawlers.utils.group_alarm import catch_except
from jinja2 import Template


class BtcArh999Spider(SpiderBase):
    name = 'idx-btc-arh999'
    url = 'https://fapi.coinglass.com/api/index/ahr999'

    def start_requests(self):
        yield scrapy.Request(url=self.url)

    @catch_except
    def parse(self, response, **kwargs):
        data = response.json()['data']
        params = {
            'arh_999': round(data[-1]['ahr999'], 2),
            'btc_price': data[-1]['value'],
            'change': round(((float(data[-1]['value']) - float(data[-2]['value'])) / float(data[-2]['value'])) * 100, 2)
        }
        today_start_time = str(int(time.mktime(time.strptime(str(datetime.date.today()), '%Y-%m-%d'))) + 1)
        value = rds.get(self.name, today_start_time)
        if value is not None:
            return
        rds.set(self.name, today_start_time, json.dumps(params), 60 * 60 * 24 * 2)
        print(Template(self.alert_en_template()).render(params))
        print(Template(self.alert_cn_template()).render(params))
    # must be declare
    def alert_en_template(self):
        return """The current BTC ahr999 (AHR Index) is {{arh_999}}. This spot is theoretically unsuitable for bottom fishing or long-term fixed investment. The current price of BTC is {{btc_price}}, and 24H  change is {{change}}. (The above content does not constitute investment advice and is for your reference only. Invest at your own risk.)
        """

    # must be declare
    def alert_cn_template(self):
        return """当前 BTC ahr999(九神指数)为 {{arh_999}}，理论上不宜买入抄底或定投 BTC。当前 BTC 现价 {{btc_price}}，24小时涨跌幅为 {{change}}。（以上内容仅供参考，非投资建议，风险自担。）
        """

```

Get value:

```python
import json

import scrapy
import time
import datetime
from crawlers.utils import SpiderBase, rds
from crawlers.utils.group_alarm import catch_except
from jinja2 import Template


class BtcArh999Spider(SpiderBase):
    name = 'idx-btc-arh999'
    url = 'https://fapi.coinglass.com/api/index/ahr999'

    def start_requests(self):
        yield scrapy.Request(url=self.url)

    @catch_except
    def parse(self, response, **kwargs):
        data = response.json()['data']
        params = {
            'arh_999': round(data[-1]['ahr999'], 2),
            'btc_price': data[-1]['value'],
            'change': round(((float(data[-1]['value']) - float(data[-2]['value'])) / float(data[-2]['value'])) * 100, 2)
        }
        today_start_time = str(int(time.mktime(time.strptime(str(datetime.date.today()), '%Y-%m-%d'))) + 1)
        # redis get method
        value = rds.get(self.name, today_start_time)
        if value is not None:
            return
        # redis set method
        rds.set(self.name, today_start_time, json.dumps(params), 60 * 60 * 24 * 2)
        print(Template(self.alert_en_template()).render(params))
        print(Template(self.alert_cn_template()).render(params))
    # must be declare
    def alert_en_template(self):
        return """The current BTC ahr999 (AHR Index) is {{arh_999}}. This spot is theoretically unsuitable for bottom fishing or long-term fixed investment. The current price of BTC is {{btc_price}}, and 24H  change is {{change}}. (The above content does not constitute investment advice and is for your reference only. Invest at your own risk.)
        """

    # must be declare
    def alert_cn_template(self):
        return """当前 BTC ahr999(九神指数)为 {{arh_999}}，理论上不宜买入抄底或定投 BTC。当前 BTC 现价 {{btc_price}}，24小时涨跌幅为 {{change}}。（以上内容仅供参考，非投资建议，风险自担。）
        """

```


# Related Links

KingData - The most popular platform that focuses on data  monitoring and broadcasting in Web3.📈

🔹 Website: <https://kingdata.com&#x20>;

🔹 News channel: @KingData\_News&#x20;

🔹 CN Telegrams Group: @KingDataCN&#x20;

🔹 EN Telegrams Group: @KingDataEN&#x20;

🔹 Twitter: <https://twitter.com/KingData\\_com&#x20>;

🔹 Discord: <https://dsc.gg/kingdata>


# Integrate KingData content into your app

Many users want to integrate KingData content into their own products, in this regard, KingData officials welcome you to integrate KingData data monitoring content, the integration steps are explained below.

*<mark style="color:orange;">Note: In order to ensure the real-time data of monitoring content, all KingData content will be broadcasted to all customers by active push. If you want to pull KingData's past monitoring content, please contact the official customer service for further information.</mark>*

**KingData monitors data content integration steps as follows:**

**Step 1:** Contact customer service Telegram [<mark style="color:red;">@byonlee</mark>](#user-content-fn-1)[^1], apply to buy API License key.

**Step 2:** Prepare the url and KingData account, provide it to KingData official to receive KingData official push content in real time.

**Step 3:** Log in to KingData.com and follow any indicators you wish to receive a push on.

**Step 4:** After receiving the data, parse the data format of the content pushed by KingData, the data format is as follows:

```

{
    "id": 368820,
    // Different languages of content          
    "content": {
    	"cn": "据 KingData 监控，BNB 突破 $240.0， 现报价 $240.5，24 小时涨跌幅 0.29%。",
    	"en": "According to KingData monitoring, BNB breakout $240.0, current price $240.5, 24H change is 0.29%."
    },
    // Link to the original article, May be empty
    "origin_url": "",
    // Related images, possibly empty
    "image_urls": [],
    // current content belongs to indicator
    "chart": {
        "id": 31,
        // chart name     
	    "name": {
	    	"cn": "主流币行情整数突破异动",
	    	"en": "Mainstream coin market integer breakout variation"
	    }
    },
    "published_at": 1687874711,
},

```

**Warmly welcome to integrate！！！**

[^1]: