Enabling Zero Auth Downloads for Node Operators at a Tenth of the Cost

https://blog.sui.io/aws-s3-cloudflare-r2-snapshot/

Validators and Full nodes running on the Sui network need to have the highest levels of reliability and uptime in order to deliver a high throughput scalable blockchain. A critical part of running a stateful application reliably is to ensure that hardware failovers can be done with relative ease. If the disk fails or another type of outage affects the machine running your Validator, there should be an easy way to migrate the Validator without having to reprocess all chain history. 

Ensuring seamless failovers are where Snapshots come in. State snapshots have two forms in the Sui network, Formal and Database. Formal snapshots contain the minimal state that makes up all of the Validator consensus information at the end of an epoch. Database snapshots are truly a full copy of a node’s database. 

Snapshots don’t have a lot of utility unless they are stored where they can be accessed easily and reliably. When starting to upload snapshots at the Sui network genesis, Amazon Web Services’ (AWS) S3 was the perfect choice as a reliable backend for storage which could be shared with the early node operators on the Sui network. Mysten Labs started hosting public snapshots on S3 which could be used by any node operator to quickly sync a Full node or Validator and introduce it to the network. 

However, the simple act of hosting an S3 bucket which allowed for public downloads of state snapshots turned out to be a more painful user experience than we expected. The format of the state snapshot necessitates that you use the AWS command line interface (CLI) for downloading the many files that are contained in a single snapshot. If you don’t have a pre-existing set of AWS credentials to plug in when calling the AWS CLI, you need to use a lightly documented incantation of the AWS CLI: aws s3 cp --no-sign-request

In addition to the user issues, Sui’s snapshots were growing at an exponential rate. As an ultra-high throughput blockchain, the amount of data Sui generates is almost unprecedented. Each time a state snapshot was being downloaded it would take hours to pull the over 800 gigabytes of data from S3 onto a node operator’s host.

For any readers familiar with the math behind AWS S3 pricing, the public good which was s3://mysten-mainnet-snapshots became expensive quickly. S3 charges you per gigabyte on data transferred out of S3. Because most operators are running nodes outside of AWS, this applies to the Mysten Lab’s snapshot bucket. We were quickly running up five figure monthly charges to host this public resource.

Enabling Zero Auth Downloads for Node Operators at a Tenth of the Cost
Serving state snapshots to Validators and Full Nodes resulted in daily egress from AWS S3 nearing 40 terabytes a day.

Cloudflare recently announced R2, an S3 competitor, which is unique in its pricing model: zero egress costs. This is a perfect fit for hosting a dataset which is regularly fetched, and a huge winning feature over S3. 

Rather than do a full migration, we chose to add R2 as an alternative source to S3, and move S3 to a Requester Pays model. S3 has great performance and features like global transfer acceleration that we did not want to give up. The major part of this migration was not adjusting our tooling to write to R2 (R2 is S3 API-compatible), but modifying the Sui application to easily read from R2. 

Supporting permissionless downloads on R2

Asking users to use the AWS CLI against R2 was not an acceptable experience; we instead wanted users to be able to point our tooling at db-snapshot.mainnet.sui.io and read the files hosted there with zero authentication required (having a zero authentication option was important to us as we want to make it as easy as possible for anyone to run a Sui Full node). 

AWS S3 request signing is a crucial aspect of interacting securely with Amazon S3 resources. When a request is made to an S3 bucket, whether it’s for uploading an object, downloading a file, listing objects, or any other operation, the request needs to be signed to ensure its authenticity and integrity. The process typically involves creating an authenticated http request by signing it with the user’s access keys (access key ID, secret access key). 

For publicly accessible files or objects, it is technically possible to bypass signing of requests for reading (but not listing) resources for most cloud providers but is not supported in the Rust object store library we are using. As such, we decided to add this support in our codebase. We wanted to add zero authentication support but give users the option to choose between restoring snapshots from S3 with signed requests (because buckets have requester pays mode enabled) or from R2 without signing. To do this cleanly, we first declared abstractions for common object store operations in our codebase:

#[async_trait]
pub trait ObjectStoreGetExt: std::fmt::Display + Send + Sync + 'static {
   /// Return the bytes at given path in object store
   async fn get_bytes(&self, src: &Path) -> Result;
}
#[async_trait]
pub trait ObjectStoreListExt: Send + Sync + 'static {
   /// List the objects at the given path in object store
   async fn list_objects(
       &self,
       src: Option,
   ) -> object_store::Result<BoxStream<'_, object_store::Result>>;
}
#[async_trait]
pub trait ObjectStorePutExt: Send + Sync + 'static {
   /// Write the bytes at the given location in object store
   async fn put_bytes(&self, src: &Path, bytes: Bytes) -> Result;
}
#[async_trait]
pub trait ObjectStoreDeleteExt: Send + Sync + 'static {
   /// Delete the object at the given location in object store
   async fn delete_object(&self, src: &Path) -> Result;
}

For cleanly switching between signed and unsigned implementations, our usage functions needed to be modified a bit as well:

/// Read object at the given path from input store using either signed or 
/// unsigned store implementation
pub async fn get(store: &S, src: &Path) -> Result;
/// Returns true if object exists at the given path
pub async fn exists(store: &S, src: &Path) -> bool;
/// Write object at the given path. There is no unsigned put implmenetation
/// because writing an object requires permissioned user signing requests
pub async fn put(store: &S, src: &Path, bytes: Bytes) -> Result;

We then implemented the signed (by falling back to the object store library we were already using) and unsigned implementations (leveraging individual cloud providers’ REST APIs) of the above traits: 

/// Implementation for making signed requests using object store lib
#[async_trait]
impl ObjectStoreGetExt for Arc {
   async fn get_bytes(&self, src: &Path) -> Result {
       self.get(src)
           .await?
           .bytes()
           .await
           .map_err(|e| anyhow!("Failed to get file: {} with error: {}", src, e.to_string()))
   }
}
/// Implementation for making unsigned requests to [Amazon 
/// S3](https://aws.amazon.com/s3/).
#[derive(Debug)]
pub struct AmazonS3 {
   /// Http client wrapper which makes unsigned requests for S3 resources
   client: Arc,
}
#[async_trait]
impl ObjectStoreGetExt for AmazonS3 {
   async fn get_bytes(&self, location: &Path) -> Result {
       let result = self.client.get(location).await?;
       let bytes = result.bytes().await?;
       Ok(bytes)
   }
}
/// Implementation for making unsigned requests to [Google Cloud 
/// Storage](https://cloud.google.com/storage/).
#[derive(Debug)]
pub struct GoogleCloudStorage {
   /// Http client wrapper which makes unsigned requests for gcs resources
   client: Arc,
}
#[async_trait]
impl ObjectStoreGetExt for GoogleCloudStorage {
   async fn get_bytes(&self, location: &Path) -> Result {
       let result = self.client.get(location).await?;
       let bytes = result.bytes().await?;
       Ok(bytes)
   }
}
pub struct ObjectStoreConfig {
  /// Which object store to use i.e. S3, GCS, etc
  #[serde(skip_serializing_if = "Option::is_none")]
  #[arg(value_enum)]
  pub object_store: Option,
  /// Name of the bucket to use for the object store. Must also set
  /// `--object-store` to a cloud object storage to have any effect.
  #[serde(skip_serializing_if = "Option::is_none")]
  #[arg(long)]
  pub bucket: Option,
  #[serde(default)]
  #[arg(long, default_value_t = false)]
  pub no_sign_request: bool,
  ...
}
impl ObjectStoreConfig {
  pub fn make_signed(&self) -> Result<Arc, anyhow::Error> {
    match &self.object_store {
      Some(ObjectStoreType::File) => self.new_local_fs(),
      Some(ObjectStoreType::S3) => self.new_s3(),
      Some(ObjectStoreType::GCS) => self.new_gcs(),
      _ => Err(anyhow!("At least one backed is needed")),
    }
  }
}
pub trait ObjectStoreConfigExt {
   fn make_unsigned(&self) -> Result<Arc>;
}
impl ObjectStoreConfigExt for ObjectStoreConfig {
   fn make_unsigned(&self) -> Result<Arc> {
       match self.object_store {
           Some(ObjectStoreType::S3) => {
               let bucket_endpoint = { };
               Ok(AmazonS3::new(&bucket_endpoint).map(Arc::new)?)
           }
           Some(ObjectStoreType::GCS) => {
               let bucket_endpoint = { };
               Ok(GoogleCloudStorage::new(&bucket_endpoint)).map(Arc::new)?)
           }
           _ => Err(anyhow!("At least one backend is needed")),
       }
   }
}

With all of the above in place, we could cleanly switch between signed and unsigned implementations based on the user provided configuration:

let store: Arc = if store_config.no_sign_request {
  store_config.make_unsigned()?
} else {
  store_config.make_signed().map(Arc::new)?
}; 

We were close to our objective of supporting zero authentication snapshot downloads but not there yet. One last challenge involved the lack of a means to list files in an R2 bucket without signing requests (it is possible to allow public, unsigned list access on S3). And we are required to list files in a RocksDB snapshot directory before downloading it. We fixed this problem by adding a MANIFEST file with all file paths during the snapshot creation process. This MANIFEST is now the source of truth for all files and their relative paths in a snapshot directory.

Final result

In the end, by allowing R2 as the default snapshot download option we were able to reduce the cost of serving these snapshots by ~70 to 80 percent, in addition to lowering the barriers for starting and failing over nodes within the Sui network.

Note: This content is for general educational and informational purposes only and should not be construed or relied upon as an endorsement or recommendation to buy, sell, or hold any asset, investment or financial product and does not constitute financial, legal, or tax advice.

Transfer to Object Available on Sui Mainnet

https://blog.sui.io/transfer-to-object-mainnet-launch/

One of the core Move features we have been hard at work on is the ability to receive on an object. Move’s new feature, Transfer to Object, makes receiving an object by another object, rather than an account address, possible on Sui. 

Prior to Transfer to Object, only an address can be on the receiving end of a transfer. Transfer to Object makes it possible to access objects that are owned by another object, essentially by enabling the transferring of child objects out to another object under a set of pre-defined fine-grained policies (“custom receivership rules”) for the receiving operation. 


The new primitives described above make Sui’s object ownership model richer, while unlocking a new set of potential uses and enhancements.

  • Kiosk
    • With Transfer to Object, many-to-one concurrent interactions with an object are now possible. As one example, an app could send tens of ticket receipts in parallel to a ticker register object. 
    • Mass and concurrent airdrops to Kiosks, or transfer NFTs or other assets from one Kiosk to another can now be built.
  • Smart contract wallet  
    • Without Transfer to Object, wallets typically conduct transfers between accounts/addresses based on private keys. With Transfer to Object, one or more onchain “account” objects can become the hub of wallet interactions, essentially an onchain smart contract wallet. Access to to-be-transferred objects can now be associated with rich and dynamic policy checks that are as arbitrary as “is today Thursday” or “does the weather Oracle say the temperature is above 20 degrees Celsius”. If these checks fail the object transfer cannot proceed. 
    • With such fine-grained policies for object transfers, one can easily imagine constructing an onchain, upgradable, highly-secure smart contract wallet with multi-sig support and with preset spending limits for different actors to prevent wallet-draining exploits. This is essentially equivalent in functionality to Ethereum Account Abstraction described in ERC-4337
    • If an address and its private key has been compromised, one can continue to access the underlying account object itself with no change.  
  • RWA tokenization 
    • The tokenization of real world assets (RWA) gained substantial momentum in 2023, in particular among institutions for capital market instruments. 
    • Enterprises and institutions are often subject to compliance and separation-of-concern requirements. The ability to implement fine-grained and flexible transfer policies and receivership policies will be essential for institutional adoption of RWA tokenization.    

There are many available developer resources for Transfer to Object, including:

In addition to being available in Devnet, Testnet, and Mainnet, Transfer to Object is also supported in the Rust SDK, Typescript SDK, and the PySui SDK. We hope you find the potential of Transfer to Object as endless as we do. Happy building!!

Get Real-Time Weather Data with the Sui Weather Oracle

https://blog.sui.io/sui-weather-oracle/

The new Sui Weather Oracle gives builders weather data for over 1,000 cities around the world and serves as a unique randomness generator, suitable for games and betting apps requiring a trustworthy random result. Consisting of a Sui-based smart contract and a backend service pulling weather data from the OpenWeather API, anyone can integrate weather data into their app.

The SUI Weather Oracle provides information such as temperature, humidity, and wind for any of its supported cities. The oracle can be used by other smart contracts or applications that need reliable and decentralized weather information for various purposes, such as travel, insurance, agriculture, gambling, or gaming.

Get Real-Time Weather Data with the Sui Weather Oracle
The Sui Weather Oracle enables weather tracking and other apps using OpenWeather API data.

However, using weather for its randomness presents an intriguing use case. Weather is one of the most unpredictable and complex phenomena in nature. Weather forecasters do an admirable job of predicting rain or sunny skies, for the most part, but the minutiae, such as specific temperatures or wind speeds, is beyond the current science to nail down. 

As a fundamental concept in many fields of science, mathematics, and engineering, randomness can be used to generate secure cryptographic keys, test hypotheses, or simulate complex systems. However, generating true randomness is not easy, as most physical or computational processes are deterministic or biased in some way. Using weather data as an oracle input provides random outputs that are uniformly distributed and independent of any previous output.

Oracle data in apps

Oracles on Sui and other blockchains serve as conduits to offchain data, including sports scores, stock prices, and weather. Displaying this data directly in an app, the most obvious use, lets builders create stock portfolio management tools, weather trackers, and real-time football rankings. Data usage in this manner can become more complex, of course, with apps using sports scores to inform fantasy leagues as one example. 

Games may be based directly on oracle data. For example, an oracle can provide the odds and outcomes of various wagers for contests such as sports matches, political elections, or lottery draws. The players can then place bets on the outcomes based on the odds provided by the oracle. 

Things get more interesting when builders use oracles offering unpredictable data to generate randomness. For example, an oracle can use real-world events, such as weather data, sports results, or stock prices, to create random outcomes for a game. Alternatively, an oracle can use cryptographic methods, such as hash functions or digital signatures, to produce random numbers that are secure and provable. The fact that the oracle data comes from a verifiable source independent from the app makes the randomness outcomes trustworthy.

Get Real-Time Weather Data with the Sui Weather Oracle
Weather data can be used as a seed for randomness to determine the results of character actions in a game. (Image source: RPGMaker)

Randomness based on unpredictable data from an oracle could be incorporated into game mechanics. For example, in a roleplaying game, an oracle can determine the success or failure of a character’s action, such as casting a spell, hacking a computer, or persuading an NPC. The oracle can also affect the game world, such as changing the weather, spawning enemies, or triggering events. The game can use the oracle’s data to create a fair and consistent randomness that is not influenced by the game developer or the player.

The Sui Weather Oracle

Oracles will enable a new generation of apps on Sui, bridging real-world data to its high-performance platform. Given the accessibility of the OpenWeather API and the usefulness of its data, a weather oracle makes for a good addition to Sui. 

Use cases

The Sui Weather Oracle is useful for many applications that require weather data, such as:

  • Randomness: Weather data can be used as a source of randomness for various purposes, such as generating random numbers, selecting winners, or creating unique NFTs. For example, a random number generator. can use the temperature, humidity, or wind speed of a specific location at a specific time as a seed.
  • Betting and gaming: Apps can use Sui Weather Oracle data to enable weather prediction wagers, weather-themed games, or weather-based rewards. For example, a game could let users bet on a city’s weather, or an app could offer users NFTs based on the weather of different locations.
  • Other use cases: Apps for insurance, travel, education, or research could use weather data. For example, apps could factor in weather data to calculate the risk of natural disasters, plan a travel itinerary, teach students about weather patterns, or aid in setting up scientific experiments.

Structure

The Sui Weather Oracle, delivering up-to-date weather information for 1,000 cities across the globe, consists of three components: an external service, an internal service, and a smart contract. The external service, the OpenWeather API, provides current weather data from various sources. The internal service is the weather oracle back-end, which fetches the weather data from the OpenWeather API every 10 minutes and updates the weather conditions for each city. The smart contract is the Sui Weather Oracle contract, which stores the weather data on the Sui blockchain and lets users access it in a secure and transparent manner. Users can also leverage the weather data for various decentralized applications that rely on the weather, such as gaming.

Get Real-Time Weather Data with the Sui Weather Oracle

Earning storage rebates

The Sui Weather Oracle mitigates the cost of storing its real-time data on the network through the storage rebate mechanism. Sui supports onchain storage in its infrastructure and tokenomics model by adding a storage fee to its gas fees. The fund created from storage fees helps reimburse network operators for maintaining the hardware to store data and process transactions. As a means of keeping the quantity of onchain data under control, deleting onchain data renders a storage fund rebate. 

The Sui Weather Oracle stores and updates the weather data on the blockchain. The admin pays a one-time fee to create the initial list of CityWeatherOracle objects, and receives a rebate for updating the weather data for each city. The rebate is proportional to the amount of data updated, and the frequency of updates. This way, the admin can store and update the weather data on the blockchain at a low cost.

Leveraging Sui Object Display

The Sui Weather Oracle uses the Sui Object Display standard to dynamically update the icon for each city based on the current weather condition, showing rain clouds or a sun, for example. The Sui Object Display standard is a template engine that allows the onchain management of the offchain display for any type of object. It uses a template string that can be substituted with the data of the object, such as the weather ID of the city. The offchain display is handled by a back-end service that serves an icon for each one of the supported cities to the Sui explorer. The icon is chosen from a predefined set of icons that represent different weather conditions, such as sunny, cloudy, rainy, and snowy.

Get Real-Time Weather Data with the Sui Weather Oracle
The Sui Object Display standard can be leveraged to serve icons representing different weather conditions. (Image by coolvector on Freepik)

Sui Weather Oracle smart contract 

The Sui Weather Oracle smart contract provides real-time and historical weather data for 1,000 locations around the world and supports minting weather NFTs based on the weather data of a city. The smart contract also uses the Sui Object Display standard, enabling the dynamic and customizable display of the weather data and the weather NFTs on the blockchain. The smart contract has four main functions: add_city, remove_city, update, and mint

The oracle::weather module defines the following:

AdminCap, a struct that represents the administrator capability of the oracle owner.

struct AdminCap has key, store { id: UID }

WEATHER, a struct that sets up a Publisher.

struct WEATHER has drop {}

WeatherOracle, a struct, represents the oracle itself. It has fields id, address, name, and description that store the oracle’s identifier, owner’s address, name, and description respectively.

struct WeatherOracle has key {
    id: UID,
    address: address,
    name: String,
    description: String,
}

CityWeatherOracle, a struct that represents the weather data for a specific city. It has fields id, geoname_id, name, country, latitude, positive_latitude, longitude, positive_longitude, weather_id, temp, pressure, humidity, visibility, wind_speed, wind_deg, wind_gust, clouds, and dt that store the city’s unique ID, geoname ID, name, country, latitude, longitude, weather ID, temperature, pressure, humidity, visibility, wind speed, wind direction, wind gust, cloudiness, and timestamp respectively.

struct CityWeatherOracle has key, store {
    id: UID,
    geoname_id: u32,
    name: String,
    country: String,
    latitude: u32,
    positive_latitude: bool,
    longitude: u32,
    positive_longitude: bool,
    weather_id: u16,
    temp: u32,
    pressure: u32,
    humidity: u8,
    visibility: u16,
    wind_speed: u16,
    wind_deg: u16,
    wind_gust: Option,
    clouds: u8,
    dt: u32
}

An init function that initializes the contract during the deployment, creating a new instance of WeatherOracle and sharing it publicly, and creating a new instance of AdminCap and transferring it to the sender.

fun init(otw: WEATHER, ctx: &mut TxContext) {
    package::claim_and_keep(otw, ctx);
    let cap = AdminCap { id: object::new(ctx) };
    transfer::share_object(WeatherOracle {
        id: object::new(ctx),
        address: tx_context::sender(ctx),
        name: string::utf8(b"SuiMeteo"),
        description: string::utf8(b"A weather oracle for posting weather updates (temperature, pressure, humidity, visibility, wind metrics and cloud state) for major cities around the world. Currently the data is fetched from https://openweathermap.org. SuiMeteo provides the best available information, but it does not guarantee its accuracy, completeness, reliability, suitability, or availability. Use it at your own risk and discretion."),
    });
    transfer::public_transfer(cap, tx_context::sender(ctx));
}

The add_city public function, which lets the owner of the AdminCap add a new city to the oracle service by providing its geoname_ID, name, country, latitude, and longitude. The function creates a new instance of CityWeatherOracle with default weather data and adds it to the oracle as a dynamic field, using the geoname_ID as the key.

public fun add_city(
    _: &AdminCap, 
    oracle: &mut WeatherOracle, 
    geoname_id: u32,
    name: String,
    country: String,
    latitude: u32,
    positive_latitude: bool,
    longitude: u32,
    positive_longitude: bool,
    ctx: &mut TxContext
) {
    dof::add(&mut oracle.id, geoname_id, 
        CityWeatherOracle {
            id: object::new(ctx),
            geoname_id,
            name, 
            country, 
            latitude, 
            positive_latitude, 
            longitude, 
            positive_longitude,
            weather_id: 0,
            temp: 0,
            pressure: 0,
            humidity: 0,
            visibility: 0,
            wind_speed: 0,
            wind_deg: 0,
            wind_gust: option::none(),
            clouds: 0,
            dt: 0
        }
    );
}

The remove_city public function, which lets the owner of the AdminCap remove an existing city from the oracle service by providing its geoname_ID. The function removes the corresponding instance of CityWeatherOracle from the oracle’s dynamic fields and deletes the object.

public fun remove_city(_: &AdminCap, oracle: &mut WeatherOracle, geoname_id: u32) {
    let CityWeatherOracle { id, geoname_id: _, name: _, country: _, latitude: _, positive_latitude: _, longitude: _, positive_longitude: _, weather_id: _, temp: _, pressure: _, humidity: _, visibility: _, wind_speed: _, wind_deg: _, wind_gust: _, clouds: _, dt: _ } = dof::remove(&mut oracle.id, geoname_id);
    object::delete(id);
}

The update public function, which allows the oracle owner to update the weather data for an existing city by providing its geoname_ID and the new weather data. The function mutates the corresponding instance of CityWeatherOracle with the new weather data.

public fun update(
    _: &AdminCap,
    oracle: &mut WeatherOracle,
    geoname_id: u32,
    weather_id: u16,
    temp: u32,
    pressure: u32,
    humidity: u8,
    visibility: u16,
    wind_speed: u16,
    wind_deg: u16,
    wind_gust: Option,
    clouds: u8,
    dt: u32
) {
    let city_weather_oracle_mut = dof::borrow_mut(&mut oracle.id, geoname_id);
    city_weather_oracle_mut.weather_id = weather_id;
    city_weather_oracle_mut.temp = temp;
    city_weather_oracle_mut.pressure = pressure;
    city_weather_oracle_mut.humidity = humidity;
    city_weather_oracle_mut.visibility = visibility;
    city_weather_oracle_mut.wind_speed = wind_speed;
    city_weather_oracle_mut.wind_deg = wind_deg;
    city_weather_oracle_mut.wind_gust = wind_gust;
    city_weather_oracle_mut.clouds = clouds;
    city_weather_oracle_mut.dt = dt;
}

Integrating the Sui Weather Oracle

Using the Sui Weather Oracle in a Move project requires adding it as a dependency in the project’s Move.toml file:

[package]
name = "..."version = "..."
[dependencies]
Sui = { git = "", subdir = "crates/sui-framework/packages/sui-framework", rev = "..." }
oracle = { git = "", subdir = "weather-oracle", rev = "db04fbd17d6ba91ade45c32f609b949fb47d209b" }
[addresses]
...
oracle = "0x8378b3bd39931aa74a6aa3a820304c1109d327426e4275183ed0b797eb6660a8"

Creating this dependency lets builders import the oracle::weather module into Move code and make use of the Weather Oracle and its functions. The Sui Weather Oracle provides real-time weather data, such as temperature, humidity, and wind speed, for different cities around the world. The city_weather_oracle_temp function retrieves the temperature of a city in Kelvin multiplied by 1,000, given its geoname_ID.

For example, the following code retrieves the current temperature of Paris, France (2988507):

use oracle::weather::{WeatherOracle};
fun get_temp(weather_oracle: &WeatherOracle): u32 {
    let geoname_id = 2988507; // Paris, France
    oracle::weather::city_weather_oracle_temp(weather_oracle, geoname_id)
}

Bridging data to Sui

The rise of big data over the last few decades has created vast repositories representing all manner of real world phenomenon and activity, from ocean currents to traffic flows in Manhattan. Responsible use of this data results in apps that make a positive impact on people’s lives, whether it’s helping predict forest fire danger or making popular concert tickets available in a fair distribution. 

The spread of oracles on Sui will increase the relevance of apps to people’s daily lives, making the network an indispensable part of modern digital infrastructure.

View Verified Move Source Code in Sui Explorer

https://blog.sui.io/explorer-verified-source-code/

Sui Explorer now incorporates a source code view for Sui framework packages, which has been a widely requested feature. This new Source Verified tab sits alongside the existing Bytecode tab, letting developers click to view source code for the following packages:

View Verified Move Source Code in Sui Explorer
The Source Verified tab in Sui Explorer shows source code for specific framework packages.

This new feature lets Sui developers easily find, read, understand, and share Move code to ease development or auditing. For supported packages, developers no longer need to track down the specific repositories or branches that correspond to published on-chain bytecode. Simply look up the source, including docstrings, for modules in Sui explorer.

Verifying the source code

This new feature verifies that it is displaying the correct and up-to-date source code for a published package through a backend service which checks that the reference source code compiles to the same bytecode on-chain. That is, the service verifies that compiling the source code locally yields the exact same bytecode. 

View Verified Move Source Code in Sui Explorer
This feature ensures accurate source code by compiling it and comparing its bytecode to the on-chain bytecode.

The system works by cloning a well-known source repository with published packages (1) and downloading the corresponding on-chain bytecode for the published packages as specified in the source repository (2). The service compiles the source code and verifies that it matches the bytecode (3). If the compiled source code matches, Sui Explorer displays the source code when requested (4). When a source repository is updated for a tracked branch, the process effectively repeats, and the server updates tracked branches at a steady cadence.

This system does not derive or decompile source code from on-chain bytecode, instead expecting an existing repository containing a reference implementation to verify against.

More packages to come

In this first phase, the feature supports source display for a handful of well-known packages (0x1, 0x2, 0x3, 0xdee9), and will support more in the future. Developers interested in contributing their own reference source code can take a look at the configuration file and consider making a pull request to track and display their Move code in Sui Explorer.

Delivering Fair Gas Fees Through Resource Usage Metering

https://blog.sui.io/computation-costs-gas-fee-model/

Sui’s massive parallel processing requires new thinking in how to apply gas fees, the cost of processing transactions on the network. In our work, we examine computation costs and instruction processing to engineer an optimal gas fee mechanism. Accurately assessed gas fees not only deliver fair network cost spreading and healthy operational business models, but encourage developers to use best practices to ensure appropriate resource usage.

Sui’s gas model leverages a Validator survey to set the gas price and ensure that the cost of on-chain transactions are disciplined by the network’s operational costs. However, defining the specific cost of a Sui transaction is complicated by the fact that each transaction includes multiple instructions, and the network can process multiple transactions simultaneously. In other words, how many gas units should each transaction cost?

Applying per transaction gas fees in the same manner as other blockchains, that can only process transactions serially, would likely overcharge Sui users. Our model involves measuring instruction count, stack depth, and memory usage to more accurately represent network resource usage. Overall, our ongoing work involves simulated and real-world testing to deliver a market-based gas price with a resource-based gas schedule to deliver an efficient gas fee mechanism serving Sui’s users and network health.

In sum, we’ve developed Sui’s gas fee mechanism through the following broad steps:

  • Determine the best way to count resource usage
  • Design a method to calculate computation costs
  • Analyze the gas fee model based on various load assumptions

Resource usage on a parallel processing network 

As blockchains are a shared resource, gas fees account for the usage of that resource, compensating validators for the operational costs of maintaining the network. In addition, gas fees affect user behavior, encouraging judicious use of the network and, hopefully, discouraging misuse.

Many blockchain languages, such as Solidity and the Ethereum virtual machine, and certain dialects of Move use bytecode-based metering approaches, associating a specific cost with each individual bytecode instruction. In these settings, the accuracy of each bytecode cost to the number of underlying computational costs (i.e., CPU cycles) is not as important as the relative magnitude of the costs of these instructions. For example, if an Add instruction has a runtime of 1 millisecond and Div has a runtime of 2 milliseconds, what matters is that gas_cost(Div)/gas_cost(Add) ~= 2 while the actual underlying numbers assigned for gas_cost(Add) and gas_cost(Div) matter less.

In more traditional blockchains, where a total order to execution is part of the protocol, it makes sense to think of a single transaction as an exclusive use of the network. In particular, if one user runs some computation another user cannot run theirs until the prior computation completes. In such a setting gas costs need to be highly reflective of the relative runtime of a transaction because transaction processing is a zero-sum game–I got time, so everyone else didn’t.

Pricing a transaction as if it was running by itself may not be accurate, and in fact would most likely significantly overcharge the user.

On Sui, however, the protocol includes massively parallel execution, making resource use non-exclusive. Pricing a transaction as if it was running by itself may not be accurate, and in fact would most likely significantly overcharge the user. Transaction processing is no longer necessarily a zero-sum game–I can have time, and thousands or even hundreds of thousands can also share that same time to process their transactions.

Sui’s parallelism caused us to take a step back and examine the traditional instruction-based approaches towards gas metering during execution, and whether they were the best fit for this new protocol. In particular, parallelism adds a fluctuation to the cost of resource usage that calls for a model where we group similar things together instead of applying instruction-specific pricing. For example, a specific cost for adding two integers, and a different specific cost for dividing two integers is less important, because the actual runtime costs are more difficult to predict due to parallel artifacts.

With this in mind, we set out to design a gas model to both capture the new parallel aspects of execution, make the resulting model as easy to understand and reasonable as possible, and most importantly to give those wins back to Sui’s users. Developing this model led us to reject the approach of adapting an inherently serial and historical bytecode-based gas mechanism that attempts to measure each computational cycle and instead design a curve or tier-based gas model for execution. 

Designing Sui’s gas model

Coming up with an accurate model for computation costs, leading to fair gas fees, required discarding what might be relatively easy paths, such as measuring transaction time, and investigating other computational factors. Along with arriving at a reasonable model, we also need to consider Sui’s storage fee, which allows for long-term on-chain storage.

Computational costs

With parallelism and efficient computation native to the Sui Move virtual machine, the comparative overhead of executing a single instruction becomes less important to the overall execution time or duration of the transaction. Instead, the sheer number of instructions executed matters the most in determining accurate overhead. Only after executing large numbers of bytecode instructions does the actual runtime cost difference for each instruction become significant to the overall runtime of the transaction.

In addition, creating a cost model that accurately reflects the real-world costs of execution, even for a serial execution system, requires overcoming a number of challenges. 

  • Complicated and highly complex gas charging mechanisms are difficult to understand, program against, and implement accurately. 
  • The gas calculation for an instruction can be nearly as costly as the execution of the actual instruction. 
  • The execution times for the same instruction may vary significantly depending on other contextual information not taken into account in the model (which is essentially impossible to model reasonably), regardless of the complexity of the gas model. The paper, Broken Metre: Attacking Resource Metering in EVM, by researchers Daniel Perez and Benjamin Livshits, explores these challenges in depth.

With these issues in mind, we set out to explore whether there was a better cost model for gas computation in Sui. We examined ways of coarsening the cost model through different dimensions of computation, rather than relying on precise and discrete counting of instructions and memory. Additionally, we aimed to keep the resulting model as simple and easy to understand as possible, while also ensuring that it is fair and accurate for users.

From this exploration, we arrived at a model where we track multiple different dimensions of computational cost simultaneously during execution based on three factors: 

  • Instruction count
  • Stack depth 
  • Memory usage 

Each of these different costs contribute to transaction resource usage in different ways, with different weights, and the cost of using one unit of each (e.g., executing a single instruction, allocating a single byte) will increase as more and more of these resources are used over the course of the transaction’s execution. The amount that each one increases, and how much of that resource needs to be used for it to start increasing, are defined independently of each other, using different cost curves. Each cost for each dimension during execution is then combined with step functions.

The core idea with this new model was that we don’t aim for precision in an exact, cycle-accurate manner, but rather define a range for which transaction costs are similar. Additionally, the individual cost of a single instruction matters less than the overall cost of executing all of the instructions of the transaction. Therefore, this new design takes a more dynamic view towards resource costing, where the more you use the resource in a transaction, the more that resource costs. We drew some inspiration from the paper, An Adaptive Gas Cost Mechanism for Ethereum to Defend Against Under-Priced DoS Attacks, by the researchers Ting Chen, Xiaoqi Li, Ying Wang, Jiachi Chen, Zihao Li, Xiapu Luo, Man Ho Au, and Xiaosong Zhang.

Dynamically pricing instructions and resource usage in this way lets us underprice resource usage under certain bounds, since we know that if those bounds are exceeded (e.g., in an attack scenario), the dynamic pricing of these resources during execution will eventually cause the computation to become too expensive to reasonably exploit. This ability to underprice resource usage at the lower end of computational usage (e.g., less than 100,000 instructions, less than 1 megabyte of total allocations) allows us to incentivize and encourage the idea of “conscious computations” while minimizing complexity, both in implementation, explanation, and reasoning.

This design makes the need to optimize code for gas cost less of an issue.

Due to the simplicity of the model at the lower levels of computational complexity (where every instruction costs one gas unit and every allocation costs one gas unit), the model is easy to understand and inexpensive. This design makes the need to optimize code for gas cost less of an issue. Because every instruction cost is the same, it is almost impossible to do so. 

In this way we use our gas model to encourage proper programming design and remove the focus on minor details, single bytecode instructions, and obtuse optimizations that can often be harmful to the program. At the same time, we wanted developers to be aware of their computations and the resources they use while still being fair in accounting for their resource usage.

Non-computational costs

In addition to the computational costs described above, there are additional costs important to the overall cost of executing a transaction on Sui. The two primary sources of this additional cost today are long-term storage costs and gas cost rounding. 

Storage costs are handled in a unique way via Sui’s Storage Fund. Once we finish executing a transaction, we compute the data to be allocated to long-term storage and this additional cost will be added to the overall cost of executing the transaction. Moreover, if someone deletes storage, a storage refund will be given that can either reduce the total cost of the transaction, or even result in a refund. However, these types of storage refunds are only assessed after the execution of the transaction and therefore cannot contribute to the overall amount of gas that can be used during the execution of the transaction. 

Gas cost rounding occurs due to the step function in gas metering, where we put computational units in buckets ranging from 1,000 to 5,000,000. Any transaction using less than 1,000 computational units would be rounded up to 1,000, resulting in a negligible additional cost.

Detailed Sui gas cost analysis

The gas cost model we designed needs to work in practice as well as theory. Testing it involved looking at transaction costs for different synthetic workloads. In particular, given N instructions, what would the costs look like for different values of allocations, or pushes and pops for each instruction in the cost model.

Synthetic case 1: Instructions with no allocation

We start by looking at the most synthetic workload of all, instructions with no allocations that also do not affect the stack. Each instruction is essentially a NoP instruction. Below we see the cost for execution on the y-axis as more and more instructions are executed along the x-axis.

Delivering Fair Gas Fees Through Resource Usage Metering
Instructions with no allocations begin to rise at certain thresholds, as opposed to a linear increase. Our model makes transactions with excessive instructions costly, which should limit resource-hungry apps.

As we can see from this test, and as was discussed previously, the cost per instruction is not linear. After a certain threshold charges will be multiplied by a constant making the final cost more expensive. The object is to keep costs contained for as much and for as long as possible while at the same time avoiding denial of services attacks or usages of the network that result in a similar effect.

This non-linear increase in costs lets us keep costs contained for most common usages of the network, while at the same time allowing heavy transactions to be executed for higher fees if and when users are willing to pay. Finally, there has to be a point where resource usage is not justified any longer and possibly penalizing the system and other users. So for certain usage and consumption of resources we want to make the cost prohibitively expensive (while not placing a strict upper bound).

Synthetic case 2: Pure allocation

In the second synthetic test case, we looked at an ideal setting where we somehow managed to allocate bytes of memory during execution while not executing any instructions. The graph below shows the cost along the y-axis for allocating up to 43 megabytes of memory. 

Delivering Fair Gas Fees Through Resource Usage Metering
The cost for pure memory allocation rises at certain thresholds, similar to our previous test case using only instructions. However, here the costs increase more quickly as memory usage grows.

Again, as we saw in the previous case the cost of allocating memory scales non-linearly as more and more memory is allocated during execution. However, as opposed to instruction execution the non-linear scaleup for memory allocation is much higher as the upper ends of the costs are reached.

Having looked at the different primary dimensions separately, we can now take a look at some more realistic (yet still synthetic) workloads.

Synthetic case 3: Instructions with different allocation amounts 

Having looked at instruction and memory costs in isolation and how costs change as more instructions are executed or more memory is allocated, we can now analyze the cost landscape when we start putting them together. The below graph shows costs on the y-axis as the number of instructions increases on the x-axis for different instruction counts.

Delivering Fair Gas Fees Through Resource Usage Metering
In these synthetic loads, the two most likely to mimic real-world use are memory plus instruction (orange) and memory plus instructions (bright blue). Gas costs increase substantially after particular thresholds.

One thing to note is how the cost is very contained for a period of time (number of instructions) and then grows exponentially from there. This gets back to what we mentioned earlier where we could handle either underpricing or aggressively pricing costs at the lower end of the computational usage scale because, as computational usage within a transaction increases, the cost of that computation will increase dramatically along with it, counteracting any possible underpricing or computational artifacts not taken into account by the cost model.

Real world testing and tuning

The tuning and evolution of the Sui gas model both for computation and for non-computational aspects of the system is an ongoing process where we need to constantly evaluate how the current model behaves against the real world, and determine where the pain points are for developers and users. We plan on continuing to develop a fair and comprehensible model that can be easily programmed against, that doesn’t discourage good programming habits as much as possible, and incentivizes good behavior. 

Looking to the future, the concept of opportunity cost and how gas fees and the gas model relate to the parallel nature of execution in Sui means that we will need to think about ways of incurring additional costs for interacting with or using shared objects that create serialized sections of computation.

Additionally, we will want to add additional costs to cover non-computational costs of the transaction, such as network bandwidth. We will look at ways that the transaction size, input objects’ count and size, and input arguments’ count and size can be factored into the overall cost of executing the transaction.

The evolving gas story in Sui is an ongoing process and we value community feedback on both places where the gas model could be improved and where tooling and greater visibility and information into how transactions use gas would be useful.

Sui Linters and Warnings Update Increases Coder Velocity

https://blog.sui.io/linter-compile-warnings-update/

New linter support and enhanced warning messages launched on Sui improve the Move developer experience, offering coding support common in many programming ecosystems. Six new linters, mostly dealing with object handling, find potentially problematic patterns in Sui specific code. In addition, the Move compiler now includes warnings about unused constructs, which should prove helpful to Move developers. 

These additions help both new and experienced developers avoid problems with their code as they build apps taking advantage of Sui’s innovative technology. Both the linters and warning messages include suppression options, letting developers customize their workflows.

Linting support

Many developers depend on linters to catch problems in their code as they work, especially in high productivity environments where code reviews might be rare. However, because Sui Move is relatively new, it hasn’t enjoyed the support of coding aids common in languages that have been around for years.

With this new framework on Sui, linting support is currently opt-in. Developers need to specify the —-lint flag to any build, test, publish, and upgrade commands to see linter messages. In a future release, this framework will switch to an opt-out mode (—-no-lint) so that linter messages display by default.

Sui currently supports six different linters, described below, with more planned. Community feedback will guide future linter development.

1. Coin field linter

This analysis flags uses of the sui::coin::Coin object in fields of other objects and structs. In most cases, developers should use sui::balance::Balance to save space.

As an example, consider this simple module defining an object containing a field of the sui::coin::Coin type:

#[allow(unused_field)] module coin_field::test {     struct S1 {}     struct S2 has key, store {         id: sui::object::UID,         c: sui::coin::Coin,     } }

Building this module results in the following linter message:

warning[Lint W03001]: sub-optimal 'sui::coin::Coin' field type   ┌─ ./sources/test.move:5:12   │ 5 │     struct S2 has key, store {   │            ^^ The field 'c' of 'S2' has type 'sui::coin::Coin' 6 │         id: sui::object::UID, 7 │         c: sui::coin::Coin,   │         - Storing 'sui::balance::Balance' in this field will typically be more space-efficient   │   = This warning can be suppressed with '#[lint_allow(coin_field)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

Note that the #[allow(unused_field)]annotation in the source code suppresses printing unused field warnings to make the output more succinct. The Enhanced warnings section of this article, below, goes into more detail about new warnings as well as warning and linter message suppression. 

2. Collection equality linter

This linter flags situations when instances such as sui::table::Table, sui::table_vec::TableVec, sui::bag::Bag are being compared for (in)equality. The reasoning behind this linter is that this type of comparison is not very useful and does not take into consideration structural (in)equality.

As an example, consider this basic module containing a function attempting to compare references to two different instances of sui::bag::Bag:

module collection_eq::test {     public fun bag_eq(bag1: &sui::bag::Bag, bag2: &sui::bag::Bag): bool {         bag1 == bag2     } }

Building this module results in the following linter message:

warning[Lint W05001]: possibly useless collections compare   ┌─ ./sources/test.move:3:14   │ 3 │         bag1 == bag2   │              ^^ Comparing collections of type 'sui::bag::Bag' may yield unexpected result.   │   = Equality for collections of type 'sui::bag::Bag' IS NOT a structural check based on content   = This warning can be suppressed with '#[lint_allow(collection_equality)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

3. Custom state change linter

This linter flags potential custom implementations of transfer, share, and freeze calls on objects that already have a store ability and where the developer can use public variants of these calls. Using these calls can be dangerous as custom transfer, share, and freeze operations become unenforceable in this situation.  A function is considered a potential custom implementation if it takes as a parameter an instance of a struct type defined in a given module with a store ability and passes it as an argument to a private transfer, share, or freeze call.

As an example, consider this simple module containing a function attempting to use the public sui::transfer::transfer function to transfer an object with the store ability passed as an argument:

#[allow(unused_field)] module custom_state_change::test {     struct S1 has key, store {         id: sui::object::UID     }     public fun custom_transfer(o: S1, a: address) {         sui::transfer::transfer(o, a)     } }

Building this module results in the following linter message:

warning[Lint W02001]: potentially unenforceable custom transfer/share/freeze policy   ┌─ ./sources/test.move:7:16   │ 7 │     public fun custom_transfer(o: S1, a: address) {   │                ^^^^^^^^^^^^^^^ - An instance of a module-private type with a store ability to be transferred coming from here   │                │                   │                Potential unintended implementation of a custom transfer function. 8 │         sui::transfer::transfer(o, a)   │                        -------- Instances of a type with a store ability can be transferred using the public_transfer function which often negates the intent of enforcing a custom transfer policy   │   = A custom transfer policy for a given type is implemented through calling the private transfer function variant in the module defining this type   = This warning can be suppressed with '#[lint_allow(custom_state_change)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

4. Freeze wrapped linter

This linter flags freezing objects containing (transitively or not) inner objects. In other words, it flags freezing of objects whose fields (directly or not) wrap other objects. Freezing such objects prevents unwrapping of inner objects.

As an example, consider this basic module containing a function attempting to freeze an object of type Wrapper containing a field of another object type Inner:

#[allow(unused_field)] module freeze_wrapped::test {     struct Inner has key, store {         id: sui::object::UID     }     struct Wrapper has key, store {         id: sui::object::UID,         inner: Inner,     }     public fun freeze(w: Wrapper) {         sui::transfer::public_freeze_object(w);     } }

Building this module results in the following linter message:

warning[Lint W04001]: attempting to freeze wrapped objects    ┌─ ./sources/test.move:13:45    │  9 │         inner: Inner,    │                ----- The field of this type is a wrapped object    · 13 │         sui::transfer::public_freeze_object(w);    │                                             ^ Freezing an object of type 'Wrapper' also freezes all objects wrapped in its field 'inner'.    │    = This warning can be suppressed with '#[lint_allow(freeze_wrapped)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

5. Self transfer linter

This linter flags transfers of an object to the transaction sender obtained from the sui::tx_context::sender() call. The goal of this linter is to encourage developers to return objects from functions rather than transferring them to the transaction sender. Returning objects from functions increases composability of functions in Programmable Transaction Blocks by allowing callers to directly use the returned object.

As an example, consider this simple module containing a function attempting to transfer a freshly created object to the transaction sender:

module self_transfer::test {     struct S1 has key, store {         id: sui::object::UID     }     public fun public_transfer(ctx: &mut sui::tx_context::TxContext) {         let o = S1 { id: sui::object::new(ctx) };         sui::transfer::public_transfer(o, sui::tx_context::sender(ctx))     } }

Building this module results in the following linter message:

warning[Lint W01001]: non-composable transfer to sender   ┌─ ./sources/test.move:8:9   │ 6 │     public fun public_transfer(ctx: &mut sui::tx_context::TxContext) {   │                --------------- Returning an object from a function, allows a caller to use the object and enables composability via programmable transactions. 7 │         let o = S1 { id: sui::object::new(ctx) }; 8 │         sui::transfer::public_transfer(o, sui::tx_context::sender(ctx))   │         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   │         │                                 │   │         │                                 Transaction sender address coming from here   │         Transfer of an object to transaction sender address in function public_transfer   │   = This warning can be suppressed with '#[lint_allow(self_transfer)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

6. Share owned linter

This linter flags making objects passed as function parameters or resulting from unpacking shareable (which are likely already owned), which would lead to an abort. A suggested pattern is to create a fresh object and share it within the same function. Typically, any object passed to a function by value is an owned object.

As an example, consider this basic module containing a function attempting to share an object passed as an argument (data flow for this object is tracked in the function):

#[allow(unused_field)] module unused::test {     struct Obj has key, store {         id: sui::object::UID     }     public fun arg_object(o: Obj) {         let arg = o;         sui::transfer::public_share_object(arg);     } }

Building this module results in the following linter message:

warning[Lint W00001]: possible owned object share   ┌─ ./sources/test.move:9:9   │ 7 │     public fun arg_object(o: Obj) {   │                           - A potentially owned object coming from here 8 │         let arg = o; 9 │         sui::transfer::public_share_object(arg);   │         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   │         │                                  │   │         │                                  Creating a fresh object and sharing it within the same function will ensure this does not abort.   │         Potential abort from a (potentially) owned object created by a different transaction.   │   = This warning can be suppressed with '#[lint_allow(share_owned)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

Enhanced warnings

Along with linters and their alerts, the Move compiler now helps developers by warning about unused constructs, including unused functions, constants, (function) type parameters, and struct/object fields. Although many developers find these warnings helpful, they might be unwelcome, particularly for code that compiled previously without any warnings. Developers who want to suppress these warnings for a specific code element can specify an annotation (of the form #[...]) for a module, a constant, a type definition, or a function.

Warning suppression example

The following code provides an example that generates the new warnings. This basic module defines an unused constant,:

module unused::test {     const UNUSED_CONST: u64 = 42; }

Building this module results in the following warning:

warning[W09011]: unused constant   ┌─ ./sources/test.move:2:11   │ 2 │     const UNUSED_CONST: u64 = 42;   │           ^^^^^^^^^^^^ The constant 'UNUSED_CONST' is never used. Consider removing it.   │   = This warning can be suppressed with '#[allow(unused_const)]' applied to the 'module' or module member ('const', 'fun', or 'struct')

The compiler returns a suggestion about how to suppress a specific warning. In this case, place the #[allow(unused_const)] annotation at the constant level as follows:

module unused::test {     #[allow(unused_const)]     const UNUSED_CONST: u64 = 42; }

Including the annotation at the module level suppresses multiple warnings of the same kind:

#[allow(unused_const)] module unused::test {     const UNUSED_CONST: u64 = 42; }

Linter suppression

Developers can suppress linter messages in a similar fashion to standard compiler warnings. A linter message includes a description of an annotation that developers can use to suppress it. The keyword used to suppress a linter warning in the annotation is lint_allow instead of allow, the latter used for suppressing standard warnings. If a developer chooses linter suppression, the compiler also prints simple statistics on how many messages and of how many kinds it suppressed.

Developing on Sui

The Move programming language arrived on the scene just four years ago, and Sui Move came even later. Although the safety and utility of the language gave it immediate popularity in the builder community, the quantity of developer tools lags behind more mature languages. Adding linters and compiler warnings, some tailored specifically to Sui Move, is just one effort among many to improve the developer experience.

However, the open source nature of Move means the entire builder community can and should play a part in improving the developer experience. How people contribute ranges from expressing a need in a forum or other venue to submitting a pull request in the repo with a new linter implementation proposal or other language improvement. Whichever way the community wants to contribute, the first step towards helping to develop Move involves engaging in the Sui forums.

Maintain Sui’s Health and Earn Storage Rebates

https://blog.sui.io/deepbook-storage-fund-bot/

Unfulfilled orders on DeepBook, Sui’s first native liquidity layer, remain on-chain, using up network storage space. These unfulfilled orders also count against DeepBook’s dynamic fields limit. Maintaining a healthy network and DeepBook’s operability requires cleaning up these old orders.

Fortunately, Sui’s storage fee mechanism rewards users for removing objects from the chain. Enterprising users can earn Sui Storage Fund rebates simply by finding and deleting DeepBook’s unfulfilled orders. These actions also free up network storage and keep DeepBook available for robust order processing.

The easiest way to find and delete unfulfilled orders involves deploying a bot to do it automatically. Check out the reference code for a maximal extractable value (MEV) bot below. This free-to-use code can be compiled as is or modified to make a custom bot.

Sui storage rebates

Sui charges a storage fee along with gas fees when a transaction creates an object on the network. That storage fee goes into the Sui Storage Fund, which uses a staking mechanism to compensate network operators for maintaining on-chain storage. This design takes into consideration that operators may join and leave the network at any time, maintaining the fund in order to support operators who join after the objects they store were created.

As with all networks, unused files and other artifacts accumulate over time and take up an increasing amount of drive space. When the available storage space becomes fully used, service and performance degrades. On Sui, increased storage demands mean operators need to increase their storage capacity to ensure a performant network.

Deleting an object from Sui, such as an expired DeepBook order, frees up the space the associated objects occupied. Because storage fees are charged up front and for the life of an object, Sui returns 99 percent of the original storage fee as a rebate in SUI tokens. Storage fees on Sui are very low, so the rebate for a single deletion is correspondingly low. Although the transaction involved in deleting an object incurs a gas fee, in many cases the storage fee rebate exceeds the gas fee, resulting in a net positive outcome.

Unfulfilled DeepBook orders

Limit orders on DeepBook, where someone sets an amount they want to spend or receive for tokens, may go unfulfilled because they don’t find a match, similar to how someone might set too high a price for an auction and receive no bids. These unfulfilled orders eventually time out, yet remain on the network. Each order is an object that uses some amount of storage.

In addition, each order uses a dynamic field, a type of field that can be added or removed from an object on the fly. DeepBook has a limit of 1,000 dynamic fields across all of its pools on Sui. If the number of active plus expired orders reaches 1,000, DeepBook reaches the limit of 1,000 dynamic fields and can no longer process any orders until the number drops below 1,000.

The code below shows how to build a reference MEV bot that clears expired orders from DeepBook. When the bot deletes an object, the storage fee rebate associated with that object goes to the address used to execute the bot, helping maintain network health and DeepBook availability while rewarding the bot creator.

Build the MEV bot

The following reference code, also available in the Sui GitHub repo, demonstrates how to create a simple MEV bot to remove DeepBook expired orders from the network.

This article includes only the sections of the code needed to create the bot. See the full code in the Sui repo.

To create a client that connects to the Sui network:

const client = new SuiClient({url: "https://explorer-rpc.mainnet.sui.io:443"});

The following line retrieves all DeepBook pools using the PoolCreated events:

let allPools = await retrieveAllPools();

This section retrieves all expired orders from each pool:

let allExpiredOrdersPromises = []; for (let pool of allPools) { allExpiredOrdersPromises.push(retrieveExpiredOrders(pool.pool_id).then((expiredOrders) => { return {pool, expiredOrders} })); } let allExpiredOrders = (await Promise.all(allExpiredOrdersPromises)).flat();

This code shows how to create a transaction to clean up all expired orders, and then get the estimated storage fee rebate using devInspectTransactionBlock:

let {rebate, tx} = await createCleanUpTransaction(allExpiredOrders); console.log(`Total estimated storage fee rebate: ${rebate / 1e9} SUI`);

In addition to the code above, the code in the repo shows how to sign and execute the transaction.

The code sample in the repo also demonstrates how to use a Helper function to retrieve all pages of dynamic fields. It then shows how to split the returned array into chunks.

The dry run transaction from the example code returns the expired orders on the network, similar to the following:

Pool d9e45ab5440d61cc52e3b2bd915cdd643146f7593d587c715bc7bfa48311d826 has 6 expired orders out of 28 orders Pool f0f663cf87f1eb124da2fc9be813e0ce262146f3df60bc2052d738eb41a25899 has 6 expired orders out of 21 orders Pool 18d871e3c3da99046dfc0d3de612c5d88859bc03b8f0568bd127d0e70dbc58be has 1 expired orders out of 1 orders Pool 5deafda22b6b86127ea4299503362638bea0ca33bb212ea3a67b029356b8b955 has 5 expired orders out of 57 orders Pool 7f526b1263c4b91b43c9e646419b5696f424de28dda3c1e6658cc0a54558baa7 has 72 expired orders out of 2925 orders

Incentivized community

Decentralized, permissionless networks such as Sui require some degree of community maintenance. Because expired orders are not owned by a specific Sui address, anyone can remove them, creating an opportunity to gain storage rebates. Although there are currently not a large number of expired DeepBook orders, this proactive guidance will help ensure a healthy DeepBook system over time as use increases.