Skip to content

Commit

Permalink
updating the readme and reflect the repo ownership change
Browse files Browse the repository at this point in the history
  • Loading branch information
KeplerC committed Apr 24, 2024
1 parent c6c0f98 commit 843dea3
Show file tree
Hide file tree
Showing 8 changed files with 16 additions and 15 deletions.
4 changes: 2 additions & 2 deletions .github/rename_project.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ echo "Description: $description";

echo "Renaming project..."

original_author="KeplerC"
original_author="BerkeleyAutomation"
original_name="fog_x"
original_urlname="fog_x"
original_description="Awesome fog_x created by KeplerC"
original_description="Awesome fog_x created by BerkeleyAutomation"
# for filename in $(find . -name "*.*")
for filename in $(git ls-files)
do
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This instructions are for linux base systems. (Linux, MacOS, BSD, etc.)
- On github interface click on `Fork` button.
- Clone your fork of this repo. `git clone [email protected]:YOUR_GIT_USERNAME/fog_x.git`
- Enter the directory `cd fog_x`
- Add upstream repo `git remote add upstream https://github.com/KeplerC/fog_x`
- Add upstream repo `git remote add upstream https://github.com/BerkeleyAutomation/fog_x`

## Setting up your own virtual environment

Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
# 🦊 Fog-RT-X

[![codecov](https://codecov.io/gh/KeplerC/fog_x/branch/main/graph/badge.svg?token=fog_x_token_here)](https://codecov.io/gh/KeplerC/fog_x)
[![CI](https://github.com/KeplerC/fog_x/actions/workflows/main.yml/badge.svg)](https://github.com/KeplerC/fog_x/actions/workflows/main.yml)

🦊 Fog-RT-X: An Efficient and Scalable Data Collection and Management Framework For Robotics Learning. Support [Open-X-Embodiment](https://robotics-transformer-x.github.io/), 🤗[HuggingFace](https://huggingface.co/).

🦊 Fog-RT-X considers both speed 🚀 and memory efficiency 📈 with active metadata and lazily-loaded trajectory data. It supports flexible and distributed dataset partitioning.

[Design Doc](https://docs.google.com/document/d/1woLQVLWsySGjFuz8aCsaLoc74dXQgIccnWRemjlNDws/edit#heading=h.irrfcedesnvr) | [Dataset Visualization](https://keplerc.github.io/openxvisualizer/)

## Install

```bash
Expand Down Expand Up @@ -50,9 +49,9 @@ torch.utils.data.DataLoader(dataset.as_pytorch_dataset(desired_episodes))
```

## Design
🦊 Fog-X recognizes most post-processing, analytics and management involves the trajectory-level data, such as tags, while actual trajectory steps are rarely read, written and transformed. Acessing and modifying trajectory data is very expensive and hard.
🦊 Fog-RT-X recognizes most post-processing, analytics and management involves the trajectory-level data, such as tags, while actual trajectory steps are rarely read, written and transformed. Acessing and modifying trajectory data is very expensive and hard.

As a result, 🦊 Fog-X proposes
As a result, 🦊 Fog-RT-X proposes
* a user-friendly metadata table via Pandas Datframe for speed and freedom
* a LazyFrame from Polars for the trajectory dataset that only loads and transform the data if needed
* parquet as storage format for distributed storage and columnar support compared to tensorflow records
Expand All @@ -62,6 +61,8 @@ As a result, 🦊 Fog-X proposes
## More Coming Soon!
Currently we see a more than 60\% space saving on some existing RT-X datasets. This can be even more by re-paritioning the dataset. Our next steps can be found in the [planning doc](./design_doc/planning_doc.md). Feedback welcome through issues or PR to planning doc!

We also note we are at a beta-testing phase. We make our best effort to be backward-compatible but interfaces may be unstable.

## Development

Read the [CONTRIBUTING.md](CONTRIBUTING.md) file.
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ pip install fogx

See [Usage Guide](./usage.md) for an overview of how to use Fog-X.

You can also view [working examples on GitHub](https://github.com/KeplerC/fog_x/tree/main/examples).
You can also view [working examples on GitHub](https://github.com/BerkeleyAutomation/fog_x/tree/main/examples).
2 changes: 1 addition & 1 deletion examples/Fog_X_Cloud_Demo.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
},
"outputs": [],
"source": [
"! git clone https://github.com/KeplerC/fog_x.git\n",
"! git clone https://github.com/BerkeleyAutomation/fog_x.git\n",
"! cd fog_x && git checkout cloud-demo-dev && pip install ."
]
},
Expand Down
2 changes: 1 addition & 1 deletion examples/analytics/dataset_organizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@

dataset._prepare_rtx_metadata(
name=dataset_name,
sample_size = 10,
sample_size = 100,
shuffle=True,
)

Expand Down
2 changes: 1 addition & 1 deletion fog_x/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -576,7 +576,7 @@ def get_step_data(self) -> polars.LazyFrame:

def get_step_data_by_episode_ids(
self, episode_ids: List[int], as_lazy_frame=True
) -> List[polars.LazyFrame] | List[polars.DataFrame]:
):
"""
Args:
episode_ids (List[int]): list of episode ids
Expand Down
6 changes: 3 additions & 3 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ def read_requirements(path):
setup(
name="fog_x",
version=read("fog_x", "VERSION"),
description="Awesome fog_x created by KeplerC",
url="https://github.com/KeplerC/fog_x/",
description="Awesome fog_x created by BerkeleyAutomation",
url="https://github.com/BerkeleyAutomation/fog_x/",
long_description=read("README.md"),
long_description_content_type="text/markdown",
author="KeplerC",
author="BerkeleyAutomation",
packages=find_packages(exclude=["tests", ".github"]),
install_requires=read_requirements("requirements.txt"),
entry_points={
Expand Down

0 comments on commit 843dea3

Please sign in to comment.