作者：马士华 发表于：2008-11-14 01:20 最后更新于：2008-11-14 12:38
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息。
http://www.hadoop.org.cn/hadoop/talk-about-hadoop-distributed-lucene/

---

Lucene是大家用的最多的开源搜索引擎。本文不探讨Lucene如何实时更新（

打算写篇：
从Hadoop的 CHANGES看项目的过程
敬请期待 哈哈

kfs 中如何实现checkpoint。在CheckPoint文件中，运用在ChunkManager中，涉及到Logger Meta Restore Relpay文件

分布式文件系统的设计需求大概是这么几个：透明性、并发控制、可伸缩性、容错以及安全需求等。我想试试从这几个角度去观察HDFS的设计和实现，可以更清楚地看出HDFS的应用场景和设计理念。
    首先是透明性，如果按照开放分布式处理的标准确定就有8种透明性：访问的透明性、位置的透明性、并发透明性、复制透明性、故障透明性、移动透明性、性能透明性和伸缩透明性。对于分布式文件系统，最重要的是希望能达到5个透明性要求：
1）访问的透明性：用户能通过相同的操作来访问本地文件和远程文件资源。HDFS可以

总的抽象基类

public abstract class FileSystem extends Configured

所有的用户代码需要用到HDFS都应该采用FileSystem对象来写

本地的实现是LocalFileSystem，分布式的实现是DistributedFileSystem

public class FilterFileSystem ex

<!--- global properties -->

<name>hadoop.tmp.dir</name>

<value>/tmp/hadoop-${user.name}</value>

<description>A base for other temporary directories.</description>

如：/

[root@master bin]# ./hadoop dfs -put /root/hadoop-0.16.0/libhdfs/ /jinjin
08/04/03 16:06:26 INFO fs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /jinjin/libhdfs/libhdfs.so could only be replicated to 0 nodes, instead of 1
没有任何的数据节点，而2个数据节点明明还在运行，相互之间也能ping通。
查看一下DFS报告
[root@master bin]# ./hadoop dfsadmin -report
Total raw bytes: 0 (0 KB)
Remaining raw bytes: 0 (0 KB)
Used raw bytes: 0 (0 KB)
% u

在Namenode中通过FSNamesystem. FSDirectory.FSImage.format();

再FSImage中的

/** Create new dfs name directory. Caution: this destroys all files

l

************************************************************/
2008-04-02 17:08:08,939 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-04-02 17:08:09,016 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-04-02 17:08:09,017 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-04-02 17:08:09,017 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs]
2008-04-02 17:08:09,310 INFO org.mortbay.util.Container: Started org.m

//JobClient与JobTracker之间的通信

类似与HDFS中的ClientProtocol，用于Client与DFS的通信

但只有JobTracker实现了该接口，JobClient并不需要实现

和

运行
./hadoop jar ../hadoop-0.16.0-test.jar dfsthroughput
CTR ＋C中断后再起的情况：
[root@master bin]# ./hadoop jar ../hadoop-0.16.0-test.jar dfsthroughput
Local = /tmp/hadoop-root/mapred/temp
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for local
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
  

[root@master bin]# ./start-dfs.sh
starting namenode, logging to /root/Download/hadoop-0.16.0/bin/../logs/hadoop-root-namenode-master.out
slave1: starting datanode, logging to /root/Download/hadoop-0.16.0/bin/../logs/hadoop-root-datanode-slave1.out
slave2: starting datanode, logging to /root/Download/hadoop-0.16.0/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
master: starting secondarynamenode, logging to /root/Download/hadoop-0.16.0/bin/../logs/hadoop-root-seconda

/**

* FSImage handles checkpointing and logging of the namespace edits.

*

*/

处理namespace编辑的日志和检查点

包含FSEditLog来维护名字空间的修改

Divide the server and client configurations

The configuration system is easy to misconfigure and I think we need to strongly divide the server from client configs.

An example of the problem was a configuration where the task tracker has a hadoop-site.xml that set mapred.reduce.tasks to 1. Therefore, the job tracker had the right number of reduces, but the map task thought there was a single reduce. This lead to a hard to find diagnose failure.

Therefore, I propose separa

http://wiki.apache.org/hadoop/GangliaMetrics
在config下创建文件hadoop-metrics.properties，内容如下：

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=@GMETAD@:8650

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=@GMETAD@:8650

有什么作用，配置信息矩阵？