⚠️ Attention && FAQ

Historical data does not support storage

Some indicators often have a lot of historical trend data, which currently do not support storage. For example: When the USDT/USDT premium is greater than 1%, notify me！ we only need to store and update the latest processed data in Redis, and do not store historical data. So you need to pay attention: when the indicator is used for the first time, there is no historical data in Redis for reference and comparison. In the logic processing, it is necessary to do a good job of data initialization. Avoid program exceptions.

Data content translation

If the data content needs to be translated in the indicators, for example, if we crawl exchange announcements, some are in English and some are in Korean, then we need to translate the content into Chinese or English, and more other language versions. This is required to use translation services. The https://deepl.com service currently used by KingData. Of course, you can also choose other services. It can be implemented according to your indicator logic.

Public method

Before writing code, you can check all the files under crawlers/utils, which abstract a large number of public methods. If you find that there is no public method that meets your needs, just create a new corresponding method or file in the directory.

Proxy server

If the crawler needs a proxy service, the proxy can be ignored during local development, and KingData has a unified proxy service. You only need to declare in the Definition.md file that you need to use the proxy service.

Exception handling

Each crawler will inevitably have various exceptions in parsing and processing data logic, so @catch_except must be declared in the parsing logic method (there is a demonstration in the Demo). After the declaration, when an exception occurs in the program, the system will automatically broadcast the exception to the system's alarm platform.

Broadcast frequency limit

Even if a processing threshold is set for each indicator, sometimes a large amount of content may be broadcast. If there are a large number of high-frequency pushes, will it cause trouble for users who follow this indicator? Don't worry about this, the KingData backend limits the overall broadcast frequency. Users will not receive a large number of pushes in a short period of time.